• Sign up or login, and you'll have full access to opportunities of forum.

Traffic optimization

Go to CruxDreams.com

fallenmystic

Tribune
There is no secret. Depending on the load on the server, $ 300-350 are required every month. In the past, I funded the forum through sales at http://www.cruxdreams.com. But the site was down for several months, so I turned to forum members for help the first time.
Now http://www.cruxdreams.com is working again, but it does not yet provide sufficient funds to fund the forum.

I had already made several attempts to reach you after my suggetion about using a CDN got ignored after your first reply, but I will try it just one more time, assuming there was some mistake in communication.

Have you read the instruction about setting up XenForo to use Cloudflare that I sent you? I perfectly understand your reluctance to give me the admin access to your server and I respect that. But I hope you could try the instruction yourself, or at least let me know how you think about it, since it's quite a simple measure that can potentially reduce the traffic cost quite noticeably. (Usually, it can help a lot, but I'll be conservative about my claim as I haven't actually read the data and relevant configurations of the site.)

I don't doubt your good intention to keep the site running for all of us, for which I've always been grateful. But I wish you hadn't ignored my offer to help deal with the traffic problem when you keep mentioning the cost and encouraging people to donate.
 
Last edited:
If Cloudfare would cut costs and possibly make the site faster and safer, that seems like something that should be looked into. Imagine if you could stay at the Dorchester for the cost of a Motel6. Instead of 90%, maybe we would be at 120%...please IM, at least have a look...
 
Have you read the instruction
I remember our correspondence. But what stops me is that the service you offer is also paid.
Unfortunately, I do not have much time, but I will deal with this issue in the near future.

P.S. I will move your post and my answer after a while to the appropriate section of the forum, where we discuss technical issues.
 
I remember our correspondence. But what stops me is that the service you offer is also paid.
Unfortunately, I do not have much time, but I will deal with this issue in the near future.

P.S. I will move your post and my answer after a while to the appropriate section of the forum, where we discuss technical issues.
I'm sorry, but this is an exact quote from the message I sent you:
I chose Cloudflare among others because they provide a free plan, so we could try it first without spending more money

And again, even if you had missed that part, you could have answered my repeated messages at least, rather than just ignoring me (I made at least 4 attempts to keep the conversation going after it went cold). I only sent you such a detailed instruction which you can try yourself because it was you who suggested it in your last and only response to my message which reads:
If this problem can be solved remotely - through me, then you can try.

By the way, I mainly use AWS (which is a paid service) myself so I had to research what could be the best free alternative for CF and how it can be integrated to XenForo which I haven't used. I chose Cloudflare because that was the best free option that I could find, not because of I was affliated to the service in any way.

I tried to believe that it was just a case of honest miscommunication. But now I know that you did it knowingly, I'm beginning to regret that I've wasted my time on this matter at all.

I wish I didn't have to quote parts from our private conversation. But when you claimed that I offered you a paid service, others may think that I was trying to sell you something or that it wasn't you who kept ignoring my messages but I who made an unreasonable proposition if I don't reply.
 
Last edited:
I regret that you took my silence due to extreme employment as an insult.
This is not true.
I was just disappointed mostly when you ignored my repeated messages for 3 months. It's how you handle it now that made me feel it like an insult.

That's why I kept bring it up even when you had ignored it, but now I lost my interest after reading your reply.
 
I had a conversation with some of the people who are concerend about the matter, and a moderator wished that we could keep this thread (I didn't start this thread, by the way. FYI, my replies to another thread got moved here for housekeeping) a place for a constructive discussion.

So, I'll repost the instruction I sent to IM here, in a hope that it may somehow prove useful to someone:
I will describe the steps I would try. And please let me know if there's anything that is not clear or doesn't work as expected:

1. Subscribe to Cloudflare free plan from here.

2. They will give you a new domain for the website. Use it to update the Xenforo config (probably `library/config.php`) as follows:
PHP:
$config['externalDataUrl'] = 'http://***CHANGE-THIS***/data';
$config['javaScriptUrl'] = 'http://***CHANGE-THIS***/js';

3. In case you have any URL to images, CSS, or JavaScript files in the Xenforo admin panel, change them as well by following the steps specified in this page (number 4 and 5. It's for a different CDN provider, but the basic idea is the same).

4. Verify the settings working by opening the website in your browser, and see if attachment thumbnails, user avatars, and other images are served via CDN (right click and copy image address, or F12 to open console, and etc.)

It's likely that some images may still look as broken, since the server is currently refusing to serve them randomly, so CDN may fail to retrieve them for the same reason. In that case, just refresh the page until every image shows up. (if it doesn't work please let me know)

If everything works as intended, users would see images loading much faster, and without a problem again. And you will see decreased traffic on the hosting server.

If the measure is applied correctly, the traffic generated by all the thumbnails, avatars, and other static resources (i.e. *.css/*.js) should be minimized to a negligible level.

I can't speak for certain how much this would help reducing the cost without having access to the cost report the hosting service provides. But considering how we saw thumbnails and avatars fail to show up during the outtage, I imagine it's at least worth a try if the suggested measure will definitely minimize the traffic caused by such resources without incurring any additional cost.

There is another measure that will most likely do wonders, both to the speed of the site and the cost it may reduce. However, it requires a bit of coding and it will expose attachments to anonymous users who browse the site. So, probably it will need some discussion among the members and moderators of the community before it can be applied.

As such, I will just link to a post which contains an instruction that can be tweaked for Cloudflare, without delving too much into details:
Using DigitalOcean Spaces or Amazon S3 for file storage in XF 2.x

(If you ever decide to try it, make sure to read the terms of the CDN service you choose. Unlike the above measure, this will make CF rely on their servers as the only origin for its contents. It will do wonders, like big attachments showing up instantly at half the traffic cost, for example. But if they decide such contents violate their terms, you may have to revert the setting which may be more troublesome than applying it in the first place.)

And there's another benefit we can get by migrating to Cloudflare which is the often requested HTTPS support. As mentioned in a relevant thread, using a secure connection is simply a must for any website that requires user login nowadays. (That's why some browsers warn you that CF is not secure as it lacks HTTPS support.)

I believe SSL support is included in the free plan, as I read this page. But if not, I'd suggest consider subscribing to their 2$/month (which is a paltry sum compared with the ~300$/month the current setup costs) plan which also include things like WAF, cache analysis, and so on which are essential tools to optimize the traffic of a website.

It looks like they provide some basic tools to mitigate a DDOS or malicious bot problem. I haven't used Cloudflare as I'm more familiar with AWS, so I don't know what level of service they provide with the free plan.

But whichever plan you may choose, I'd strongly suggest to look into such tools as well because it's often the case a few malicious people or bots can generate a huge amount of traffic, if left unchecked.

Especially on a site like CF, it's pretty common to see scraping attempts to crawl the whole contents of the website so they they can be downloaded to a PC or even another server. And it is quite a trivial thing for anyone who may object to the contents we deal with on our community to run a few scripts to saturate the capacity of the server by flooding it with bogus requests.

It is important to monitor and prevent such attempts when it comes to optimize the traffic of a website, and it's not that difficult with such tools that hosting services like Cloudflare or AWS provides nowadays.

This is all I can say about the matter, without being able to access the relevant data or get feedback from those who can.

Try it or not, it's not up to my decision. But I at least tried to help the community I love as best as I could.
 
Last edited:
Glad to know that it won't be an wasted effort after all.
I'm sorry. I don't understand the benefit here exactly. (If this sounds too naive, just ignore it.)
Wikipedia says that Cloudfare is a "reverse proxy". That seems to mean that it sits between the present CF server and the users, and makes all the requests to the server itself, then returns the data to the users. So, the CF server is still there, and the requests from users are still generating web traffic, so there will still be costs for that. Is the cost minimized because the Cloudfare proxy is the only "user" the CF server sees, so the web traffic is somehow magically reduced because the data path is simpler (I am not sure how that works, because as I understand it there is never a single "data path" on the internet--the path is decided on the fly based on traffic on the routing nodes)? Thanks in advance.
 
I'm sorry. I don't understand the benefit here exactly. (If this sounds too naive, just ignore it.)
Wikipedia says that Cloudfare is a "reverse proxy". That seems to mean that it sits between the present CF server and the users, and makes all the requests to the server itself, then returns the data to the users. So, the CF server is still there, and the requests from users are still generating web traffic, so there will still be costs for that. Is the cost minimized because the Cloudfare proxy is the only "user" the CF server sees, so the web traffic is somehow magically reduced because the data path is simpler (I am not sure how that works, because as I understand it there is never a single "data path" on the internet--the path is decided on the fly based on traffic on the routing nodes)? Thanks in advance.
The important thing to note is, there are way more number of requests the users make when they browse a website than a few CDN servers can generate in the same time period.

For instance, let's assume we have a particular image on the front page of CF. It will be read from the CF's server everytime a user opens the site in a browser. And if we have 100 users visiting CF in a minute, it means the image will be requested at least 100 times a minute, incurring traffic cost each time.

But if we put a CDN in between them, the CDN server will request the image from the CF's server when it's first requested by any user, which will incur traffic cost as you said. However, the subsequent requests from any other user to the CDN won't reach the CF's server until the server cache expires. It means, it will incur zero traffic cost after the initial request is made, even there are 100 users requesting the same image as before.

As such, it can sometimes do wonders in solving a traffic problem, especially when the CDN in question offers a free plan like Cloudflare does. Moreover, there is one more benefit of using a CDN in this way.

Without a CDN, the response time and speed of loading resources from a website depends on where the server is located. For instance, if you live in Europe while the server resides in San Francisco, it's likely that the site in question would load slower, compared with the speed experienced by those who live in California.

But a CDN usually has server farms in key locations around the world, so it automatically chooses the closest location to you when it serves requested resources. So, it will also make those images and other resources delegated to it load faster than before.

However, I advise you not to get too excited too soon because there are a few factors we should consider which may also potentially affect the effectiveness of the solution in this case.

First off, the first step I suggested in the above instruction only applies to the thumbnails and avatars, not the attachments themselves. As such, if you click a thumbnail to load the full-sized image, it will still get requested from the origin as before, thus incurring the same amount of traffic.

It is because a CDN doesn't take account of security by default, so it takes additional steps to apply it over protected resources (meaning you need to login in to see them) like CF attachments. That was why I also included a more aggressive measure which can be considered in case the first step may prove insufficient.

That being said, offloading just thumbnails and avatars will at least reduce the traffic at no additional cost. In a site like this, it's typically the case that there are many more users who browse a thread (thus, requesting thumbnails and avatars) than those who opens an attachment. So, it can still be quite effective even a CDN only covers the former cases.

There is another factor which may affect the effectiveness of the measure, which is the ratio of returning users to those who browse the page for the first time. It is because, a browser also makes a local cache for resources it opens once. As such, if all users open the same page once and reload it, it would make little difference to the traffic whether or not it is served via a CDN.

As such, I assume the impact would be marginal for avatars I suppose we have many regular members whose browser likely contain a cache for avatars from other regular members. However, they will still get the benefit when they browse a thread containing thumbnails unless they reload the same post over and over rather than choosing those they haven't read yet.

It was because of such possibilities that I was reluctant to make any bold claim about the expected performance gain of introducing a CDN. I may be able to make a better prediction if I can see the log or traffic report of the server. But without such an aid, I'd like to ask to keep your expectations in check, in case that it wouldn't solve the traffic problem once and for all.

The bottom line is, it is a relatively simple measure which wouldn't cost any additional money to try that will definitely reduce some amount of the traffic our server currrently sees. The actual amount may range from marginal to very significant, but I suppose it's something worth a try regardless.

Hope this could serve to clarify the suggestion I made about migrating to a CDN to deal with the traffic problem.
 
Last edited:
I set up a forum for the cloudflare service:comp4:
Let's hope that the traffic bill at the beginning of next month will be less than in the previous one :eusa_dance:
Thanks for trying the suggestion. However, I'm afraid something is still not setup properly. If you try the verification method I mentioned in 4, you will see that the URLs for the avatars and thumbnails still refer to cruxforums.com:
4. Verify the settings working by opening the website in your browser, and see if attachment thumbnails, user avatars, and other images are served via CDN (right click and copy image address, or F12 to open console, and etc.)

1599178579900.png

I got an unwanted bill for traffic one time, but I was doing 85 mph. . .

:car:
Use a CDN! :p
 
something is still not setup properly
I used the free cloudflare plan
I have not received an offer to replace the domain.
I was asked to change the DNS domain for the forum.
I did it. There were no other instructions or suggestions.
After that, I received a message from cloudflare that everything is optimized and works fine.
 
I used the free cloudflare plan
I have not received an offer to replace the domain.
I was asked to change the DNS domain for the forum.
I did it. There were no other instructions or suggestions.
After that, I received a message from cloudflare that everything is optimized and works fine.
There are two ways to integrate CDN support for a website. The first one is to change the URL prefix for the resources (images, stylesheets, etc) so that only requests to them can be redirected to a CDN server. This was the method explained in my instruction and IIRC, all the other documentations I could find on the internet took the same approach.

However, there is another way which is to change the DNS of the whole website so that redirect every request to a CDN server. It is potentially a better way since it doesn't change anything on the user-facing side and I assume this was the method they told you to use. There is a little caveat though which is the fact that it makes it a bit difficult to see if CDN is working or not.

Fortunately, it appears that Cloudflare provides a similar way to verify if the cache is working as AWS does with which I'm more familiar. You can examine the headers of a request to such a resource in your browser's debug console to see if it's served via CDN as below:

화면 저장_20200905_062352.gif

In short, yeah it looks like it's working! :) The important thing to see here is "CF-Cache-Status: HIT" which means the image is served by a CDN server, hopefully located closer to where you live. So, this request didn't generate any traffic to CF's server which was what we tried to achieve in the first place. If it said "CF-Cache-Status: MISS", in comparison, it means that it wasn't cached from the CDN server, so it had to fetch it from CF instead, which will incur additional traffic.

So, it's better to monitor the ratio of 'hits' to 'misses' and try to maximize it and I believe Cloudflare would provide certain tools to monitor that metric somewhere in their admin page.

On a side note, remember this only caches avatars and thumbnails so the improvement potentially can be small? That was why I also mentioned a more aggressive method you can use to cache actual attachments (which must be, by far the most significant source of traffic generated on CF) themselves.

Seeing that now the basic setup is working properly, and even without having to tweak XenForo settings, now I begin to wonder what if we could cache attachments - which must be by far, the most significant source of traffic on CF - as well?

So, I'd like to ask everyone this question: should we require login to access attachments (i.e. images or movies in a post)?

If we can live with our attachments exposed to anonymous access, now there's a chance - I can't gaurantee, but it's likely - that those images can be made to load much faster than they do now, and without incurring server cost.

With avatars and thumbnails already cached on the CDN, it would most likely solve the traffic problem, not just mitigating it. So I feel this could be too good a chance to pass by without at least discussing of it, and I welcome your opinions.

@ImageMaker And thanks for your work to get it working. Now I'm glad that my suggestion was not in vain, although how much it can help must be seen when we get the bill next month.
 
Last edited:
Just one thing I forgot to mention. If you change your avatar, now it may not be immediately visible but the old one may still appear for a while. This is because now avatars are cached on the CDN server and only get periodically flushed.

So, you'll have to wait until the cache is expired. I think it's a small price to pay to optimize the traffic. But in case this is not acceptable, we can consider tweaking cache duration from the CDN console.
 
should we require login to access attachments (i.e. images or movies in a post)?
yes

at present, it is possible for 'guests' to see full-size images (uploaded to the site or externally hosted) without registering or logging in, though they can't open thumbnails or attached files, or view videos.
 
Back
Top Bottom