Problem With Images

[malins]

As we are speaking about technical stuffs now, I have to wonder if we can try using a CDN to (hopefully) mitigate the problem. I don't know XenForo well, but it's highly likely that such a popular commercial forum software would offer a some form of integration option with CDN providers.

In non-technical terms, CDN is a caching service for resources like images or javascripts and it usually runs multiple servers across the globe to automatically select the one closest to each user's location.

It can not only make loading a website much faster, but also can reduce the traffic to the original server significantly, because only non-cached resources will be pulled from the origin.

CDN usually costs money, but as it can potentially reduce the traffic by a significant margin, it can save the cost paid for the bandwidth.

A problem here is going to be that using a CDN will require a continuous and maintained increase of cash flow. Not just a one time funding effort but continuous.

Off the cuff switching the server to HTTP/2 instead of HTTP/1.1 might be a thing

Also there are a lot of small, individual HTTP requests to static files (like emojis, avatars). For each of these, cookies are passed along in the request headers, but the cookies have no functionality for a static file. A low end trick here is to simply use a separate 'cookie-free' subdomain exclusively for serving static content - something like 'static.cruxforums.com' - this would include all the things that can be seen without having a login (and that is ... thumbnails, logos, avatars, emojis). You can do this just by configuration without requiring a bigger VM.

How to Use Cookie-Free Domains - KeyCDN Support

To use cookie-free domains involves separating the content that doesn't require cookies from the content that does, therefore reducing network traffic.

www.keycdn.com

 

[Barbaria1]

Sorry to be picky , but Xenforo is just a software, and unless they also provide a hosting service, the server could be controlled by a different company.

Normally, users of such a hosting service have a control over at what capacity their website begin to fail or throttle the requests. It's just that it costs more to get a high capacity service plan.

Using a CDN, on the other hand, will reduce the amount of requests directed to the hosting service, because they will be first served by a CDN server closer to each user's location.

That is probably the case, but whatever it may be called, our site is hosted by a service which collects a fee for hosting.

 

[Nyghtfall3D]

Sorry to be picky , but Xenforo is just a software, and unless they also provide a hosting service, the server could be controlled by a different company.

That is probably the case, but whatever it may be called, our site is hosted by a service which collects a fee for hosting.

Xenforo doesn't provide hosting, they only develop the the software that powers the forum. The forum is hosted by a separate company that collects hosting fees from ImageMaker.

 

[fallenmystic]

That is probably the case, but whatever it may be called, our site is hosted by a service which collects a fee for hosting.

That is true, indeed. But the important point in this aspect is that the hosting cost could vary significantly depending on how you configure your service. For example, just forgetting to enable HTTP compression, or DDoS prevention can easily triple the bandwidth cost.

(On a side note, I believe we didn't enable the compression either, but in this case, enabling it won't probably make the problem disappear since it's probably the images that we upload to the server which is contributing to the issue the most. And most image formats are already well compressed, so doing it again will likely yield a marginal improvement. But enabling that option can still shave off a few percentage of the traffic with negligible adverse effect, so it's also something I'd like to try, if I can give such a feedback.)

There are many different ways to optimize the hosting cost, and using a CDN could be an option in many cases.

 

[Barbaria1]

I have learned a lot about things that are quite beyond me by following this discussion. We should keep in mind, though, that Cruxforums is owned and operated by ImageMaker, who has done all of us here a tremendous service over a great many years by establishing and maintaining the site, largely at his own expense. While we can identify things that might be changed or improved, and some of that might be helpful to him, it is his site to do with as he pleases.

 

[fallenmystic]

I have learned a lot about things that are quite beyond me by following this discussion. We should keep in mind, though, that Cruxforums is owned and operated by ImageMaker, who has done all of us here a tremendous service over a great many years by establishing and maintaining the site, largely at his own expense. While we can identify things that might be changed or improved, and some of that might be helpful to him, it is his site to do with as he pleases.

Of course, I agree with that 100%. l much appreciate ImageMaker for all his efforts to provide us such a rare hideout for us perv... er... those of us with special kinks, I mean.

And when I suggest something that could be tried to mitigate the problem, I don’t do it to brag my knowledge or criticize anyone of their incompetence. Rather, it’s just my way of “donating” something to help the community, as I cannot do it by the usual mean since I’m undergoing a severe financial trouble of my own.

So, I fully understand that it’s entirely ImageMaker’s call to either try or ignore my suggestions. They are just the things that I would do in a similar situation myself (I work as a programmer and manage a cloud environment) which may or may not be effective in dealing with the problem.

 

[malins]

And when I suggest something that could be tried to mitigate the problem, I don’t do it to brag my knowledge or criticize anyone of their incompetence. Rather, it’s just my way of “donating” something to help the community

That's important to understand, ... in a tech-type problem solving domain, like when you're dealing with software devs, people will throw their suggestions out openly and they can be examined and shot down or put to the test and be confirmed or rejected. None of these suggestions are attacks or criticisms, they're things that might be worth examining because they've helped in similar circumstances.

 

[Eulalia]

Yes, we appreciate that all who are contributing to this discussion are doing so with the best of intentions.
It is, I have to say, unlikely that ImageMaker will read or be able to understand what is being suggested.
It would be helpful to have specific proposals in as brief, plain and simple, non-technical English as possible,
which we can pass on to IM, that he can autotranslate into Russian.

 

[fallenmystic]

It would be helpful to have specific proposals in as brief, plain and simple, non-technical English as possible, which we can pass on to IM, that he can autotranslate into Russian.

It would be just "Let's try a CDN" then.

Seriously though, while it's not that difficult to choose and subscribe to a CDN provider, it would involve doing some technical tasks on the both ends (the CDN provider, and the Xenforo), however.

I would be even willing to volunteer to help the setting myself, if such an arrangement is possible. I know that it's generally unwise to give an admin access to a random user who joined the community just a few months ago. But maybe if you just backup everything and give me the necessary privilege on a temporary basis, something like that could be done without too much risk?

I don't claim to be the most knowledgeable guy among us on such matters. But I think it's safe to say that I know enough to perform the task, considering my job as a software engineer. Even though I'm mostly a programmer than a server engineer, I have more than 20 years of experience during which time I've built and managed similar services several times.

For the current job, I've built and managed the whole server-side infra on AWS which includes setting up a CDN service (CloudFront), for example. So, I think I won't have too much trouble trying to apply a CDN for our server, even though I haven't used XenForo before.

I know that it's a rather unusual offer, but it's just that I love this community much to be willing to help anyway I can and I feel it to have been long enough that we suffer from the image problem.

Please let me know, in case something like this could be arranged.

 

[malins]

A thing to start with might be to go through the server logs.

@fallenmystic would you agree nginx throwing 503's is mostly reacting to overload?
Question is where an overload is coming from.

The naive assumption is "there are more users visiting so the site is overloaded".
It's not always so, though. There can be rogue bots out there etc. I have some practical experience with that from a non-profit I support, who went down due to bad bots, and all you had to do was ban some malicious user agents and IP ranges and they were good to go again.

And I think it's way easier to agree to first have someone go over the nginx logs, than give any kind of admin access.
Ideally one would compare older logs (from before the problem) to newer ones but we don't know how long they are kept.

 

[fallenmystic]

A thing to start with might be to go through the server logs.

@fallenmystic would you agree nginx throwing 503's is mostly reacting to overload?
Question is where an overload is coming from.

The naive assumption is "there are more users visiting so the site is overloaded".
It's not always so, though. There can be rogue bots out there etc. I have some practical experience with that from a non-profit I support, who went down due to bad bots, and all you had to do was ban some malicious user agents and IP ranges and they were good to go again.

And I think it's way easier to agree to first have someone go over the nginx logs, than give any kind of admin access.

Yes, I think it's the most likely (not the only, however) reason for having 503 errors. And I also agree about the possibility of the problem being caused by such requests and that's why I suggested a similar thing in one of my previous posts.

However, I'd suggest just moving to a CDN instead of spending too much time on analyzing the logs, because most CDN providers have some form of DDoS protection and WAF, which is exactly the way to deal with such a problem. It's often much easier to see if and how many such requests exist using a dedicated UI for the task than doing so by reading the logs.

And even if the problem could be solved by a different mean than using a CDN, I think it'd be still good to use one for a website like this. It doesn't normally cost too much money (CloudFlare even provides a free plan) and who wouldn't love to see attached images load blazingly fast?

 

[malins]

It's often much easier to see if and where such requests are coming from using a dedicated UI for the task than doing so by reading the logs.

haha I will not argue against that

CDN sure is the right idea but there will need to be an idea how to recover cost.

Also the question remains ... why did it work without a CDN until a few months ago, and now it doesn't work anymore.

I'm just not convinced that the user base, or use intensity, has increased by say an order of magnitude.

 

[Barbaria1]

I'm just not convinced that the user base, or use intensity, has increased by say an order of magnitude.

Is it possible that capacity was simply dialed back somehow to cut costs?

 

[Jollyrei]

Also the question remains ... why did it work without a CDN until a few months ago, and now it doesn't work anymore.

Yes indeed - the first question is "what changed?". If the server host or Xanforo changed something, that might also point to a "terms of service" issue. If you pay for something, the question might be raised about whether you are getting what you paid for, if service changes arbitrarily.

 

[fallenmystic]

haha I will not argue against that

CDN sure is the right idea but there will need to be an idea how to recover cost.

Also the question remains ... why did it work without a CDN until a few months ago, and now it doesn't work anymore.

I'm just not convinced that the user base, or use intensity, has increased by say an order of magnitude.

If it was indeed caused by bandwidth outtage, I think there can be several potential explanations.

Many hosting providers impose a limit on bandwidth depending on the subscription plan, in which case, it would appear perfectly normal until the traffic reaches that threshold, but it will begin to refuse some requests once it's been surpassed.

Normally, it would only affect small percentage of those requests so you have a point to suspect there could have been a sudden burst of traffic in recent months. But there can be another explanation that it wasn't caused by intentional traffic limit, but the daemon being overloaded beyond its capacity, in which case all requests would be affected.

In either case, I believe moving to a CDN could be an effective solution. Even if it was caused by a localized attempt to DDoS the server, for example, it would be quite difficult to identify that by reading the access logs alone, and it would probably require just as much work to apply ban to the IP range than to change it to delegate static contents loading to a CDN from the XenForo/Ngnix's end.

Moving to a CDN would provide such benefits to be specific, in this case:

• Definitely making images to load much faster. (I'm sure you'd all love this part )
• Potentially reducing the traffic cost - generally speaking, a CDN charges less than ordinary hosting services for the same amount of traffic.
• Providing a tool to identify and deal with malicious bots and DDoS attacks.
• Some CDN services even provide a free SSL certificate which can be used to enable HTTPS on the server, which is a task that many of us have been asking for.

That's why I suggested it to be tried as the first solution before other things.

 

[The Fallen Angel]

It would be just "Let's try a CDN" then.

Seriously though, while it's not that difficult to choose and subscribe to a CDN provider, it would involve doing some technical tasks on the both ends (the CDN provider, and the Xenforo), however.

I would be even willing to volunteer to help the setting myself, if such an arrangement is possible. I know that it's generally unwise to give an admin access to a random user who joined the community just a few months ago. But maybe if you just backup everything and give me the necessary privilege on a temporary basis, something like that could be done without too much risk?

I don't claim to be the most knowledgeable guy among us on such matters. But I think it's safe to say that I know enough to perform the task, considering my job as a software engineer. Even though I'm mostly a programmer than a server engineer, I have more than 20 years of experience during which time I've built and managed similar services several times.

For the current job, I've built and managed the whole server-side infra on AWS which includes setting up a CDN service (CloudFront), for example. So, I think I won't have too much trouble trying to apply a CDN for our server, even though I haven't used XenForo before.

I know that it's a rather unusual offer, but it's just that I love this community much to be willing to help anyway I can and I feel it to have been long enough that we suffer from the image problem.

Please let me know, in case something like this could be arranged.

Hi there fallenmystic. Sadly moderators do not have any magic wands but I do strongly recommend that you contact ImageMaker via email with your kind offers of assistance and technical knowledge. His English is very good but as Eulalia said please keep it straightforward. The image problem does need to be sorted.

 

[malins]

Is it possible that capacity was simply dialed back somehow to cut costs?

Logically, yes it is, there's the problem that providers sometimes oversell their capacity.

That is, ImageMaker might be paying for the same service as before but if the service provider has run into problems or gone rogue they may not be fulfilling.

There's of course also the possibility that some aspect of the system has changed in way that causes it to use a greater amount of resources just to do the same thing as before. Most of us probably know this from some desktop software upgrade where the new version is just slooooow, this can happen server side too (memory leaks, etc) but one would just need to be able to monitor the health of the system.

It's impossible to diagnose this really by remote wild ass guess.

 

[fallenmystic]

Hi there fallenmystic. Sadly moderators do not have any magic wands but I do strongly recommend that you contact ImageMaker via email with your kind offers of assistance and technical knowledge. His English is very good but as Eulalia said please keep it straightforward. The image problem does need to be sorted.

Thanks! I asked it here first because I thought it would be better if the offer could be through the moderators. You wouldn't normally give admin access to your server to a random user who contacts you via an email, but if it was suggested by the moderators you trust then I thought it would have a better chance.

At any rate, if you think it'd be the best course of action, then I'll definitely send him an email. And I'm pretty sure his English would be better than mine, at the very least.

 

[malins]

• Potentially reducing the traffic cost - generally speaking, a CDN charges less than ordinary hosting services for the same amount of traffic.
• Providing a tool to identify and deal with malicious bots and DDoS attacks.
• Some CDN services even provide a free SSL certificate which can be used to enable HTTPS on the server, which is a task that many of us have been asking for.

Now if the bold bit turns out to be true, that's the winning proposition

 

[fallenmystic]

I decided to give it a go, but realized I don't know ImageMaker's email address

Can I just send him a private message instead?