Faster Webserver Responses with Browser Caching
If you have the impression that your Arduino webserver is to slow or your requests cause to much traffic, take a look how browser caching can help you to reduce webtraffic between your server and your browser.
I will cover following topics:
- How to activate a simple browser caching (Arduino UNO)
- a more sophisticated way using ETag (Arduino MEGA)
- How to use the ETag on a ESP8266 webserver
A Simple Browser Caching with Cache-Control
Every time you update the page in your browser (F5), the browser will request all files from your webserver:
Beside dynamic content you might have rather static files served on the webserver. When these files are not changing often you might consider to activate the browser cache for the individual files. This might be a stylesheet (CSS), a JavaScript or pictures. If you a very static HTML - for example a help page or a contact page etc you could use the browser cache for this HTML also.
When you have (nearly) finished your development you can add a simple line to your client output to enable the browser cache. In the HTTP header you add the field "Cache-Control". You can add the information about how long a cached version should be used. The parameter max-age takes seconds. For example to activate caching for 31 days, you can set:
message.print ("HTTP/1.0 200\r\n"
"Content-Type: text/css\r\n"
"Cache-Control: public, max-age=2678400\r\n" // cache for a of 31 days
"\r\n") ; // empty line as end of HTTP header
After adding this line you will see, that the browser is using its internal cache.
This will spare network traffic. Additionally this will speed up the time needed for the page to be displayed because the browser is using cached data and not requesting from your Arduino. A true Win-Win.
Most of the browsers will allow to overrule the internal browser cache, either with a simple "F5 reload" or a combination of "SHIFT F5". So if you update your sketch and the browser is still using the outdated cached version, you can force the browser to get the updated file from your webserver.
The HTTP Header fields Entity Tag (ETag) and If-None-Match
When a client requests a page (resource) from the server, the server can add an additional HTTP header field called "entity tag" or ETag. This ETag can be a hash of the page, a version number or something similar generated by the webserver. The content of the page will follow as usual after the HTTP body. If you check the communication between your browser and a webserver you will see following:
The browser can display the page and store it to its internal cache. If you enable the development tools in your web browser you can see this additional HTTP field.
If you request this page again, the web browser will add the former received value from the ETag to a header field named "If-None-Match". Based on the value the webserver can decide, that the web browser has a valid version of this file and therefore just notifies the browser that the file wasn't modified and it is save to to use the cached file. Even there is some TCP/IP traffic it is much less than transmitting the same file over and over again:
If you update your webserver (the hosted file), you should also change the ETag. On the next browser request, the webserver will receive the expired value from the browser. Therefore the webserver can send the new updated file (with a new ETag):
The ETag is a more sophisticated way to invalidate the browser cache. There are several advantages using the ETag:
- you have full control over when a page expires / is getting old and you can force the browser to show only valid pages
- reduced TCP/IP traffic, causing faster loading times if the cached file can be used
- no need for a manual forced (SHIFT-Reload) to get the latest version of a file.
To bring this functionality on an Arduino is not as easy than the simple "Cache-control". The ETag is a more sophisticated way to invalidate the browser cache and you must do several steps:
- add an ETag on the server responses
- check incoming request whether they contain a If-None-Match
- compare the If-None-Match with the current ETag and either respond with 304 Not Modified or with the updated file.
Some Pitfalls with the ETag
No fire without smoke. Before we talk about about the implementation you should be aware of some pitfalls with the ETag:
- The value of the ETag has to be sent in double quotes.
- Even if you send the ETag with double quotes you can't rely that a browser sends back the value in double quotes. I see several browsers which are NOT returning the ETag in double quotes (for example the OEM Samsung browser) which makes the server comparison more difficult.
- Don't rely on that you will receive the HTTP Header field exactly as "If-None-Match", be prepared to get it in small or capital letters also.
- RFC-7232 states that ETags should be content-encoded. As you will not do a GZIP on the Arduino add -a to the ETag.
Due to the different micro controllers I split the this tutorial now in Arduino AVR based boards (UNO, Mega, NANO...) and ESP8266 (NodeMCU, Wemos D1).
ETag implementation for the Arduino UNO/MEGA/NANO
I highly recommend to avoid the usage of the Arduino String class on the Arduino AVR based boards. I have shown already how to write a webserver on the UNO (MEGA or NANO). So when you implement ETag on a basic Arduino read about the general webserver and the webserver optimizations.
ETag implementation on the ESP8266
The ETag implementation on the ESP8266 is quite simple. On the ESP8266 Webserver you have for usual several definitions of handlers in your setup():
server.on("/8.htm", handlePage);
server.on("/c", handleCommand);
server.on("/f.css", handleCss);
server.on("/i.css", handleInternalCss);
server.on("/favicon.ico", handle204);
server.on("/setting.svg", handleSvg);
server.on("/clock.svg", handleSvg);
server.on("/.js", handleJs);
Due to memory restrictions the ESP8266 Webserver doesn't process all HTTP header fields. Therefore you have to add following lines in setup() - best before the server.begin() - to have access to the HTTP header fields(s). Easiest way is to define an array of char strings and handover it to the method collectHeaders() even if you are only interested in this one HTTP header field:
const char* headers[] = {"If-None-Match"};
server.collectHeaders(headers, sizeof(headers)/ sizeof(headers[0]));
I suggest to define a separate function so you can use ETag in several handlers. See following example:
/* returns true if the actual Etag is equal to the received Etag in If-None-Match header field */
bool checkETag(const char* etag) {
for (int i = 0; i < server.headers(); i++) {
if (server.headerName(i).compareTo(F("If-None-Match")) == 0)
{
String readed = server.header(i);
readed.replace("\"", ""); // some browsers (i.e. Samsung) discard the double quotes
if (readed.compareTo(etag) == 0) {
server.send(304, "text/plain", F("Not Modified"));
Serial.println(String(F("[server] ")) + server.headerName(i) + F(": ") + server.header(i) + F(" - Not Modified"));
return true;
}
}
}
server.sendHeader("ETag", String("\"") + etag + "\"");
// server.sendHeader("Cache-Control", "public, max-age=2678400"); // cache 31 days
return false;
}
The function will take the actual ETag as parameter, and will return a boolean true (mapped) or false (if not mapped) based on the comparison.
The activated HTTP header files are available in an array, therefore we have to run through the array till we find the "If-None-Match".
Based on the compare result (readed.compareTo) you either send the 304 Not Modified and leave the function with result true (=mapped) or just add a new HTTP header field "ETag" for the next Browser Response. In this case you return false (not mapped).
On the top of your file handler, you just add following lines:
const char * version {"42abc-a"};
if (checkETag(version)) return;
Obviously, "42abc-a" is your current ETag. You can use any other versioning system you have already in place. Just remember to update the versioning if you change your file.
The if statement will call the defined function. If you get a true from the function you know that the response was already 304 and therefore you don't need to process the rest of your handler. Therefore you return out of your handler back to the code which called the handler. Transmission is done!
Combine Cache-Control and ETag?
You might ask, if you can combine the Cache-Control feature with the ETag, and the answer is: Yes, you can!
It's not Working at all!?!
I struggeld with the ETag caching also, therefore here are some advices:
- Check the server response in your browsers development tools. There you can see all files sent to your browser and if they were served from cache, how long it took and more details about the TCP traffic.
- You should be able to see the HTTP header fields from your server: Does the server response contain the ETag value in double quotes?
- Does your browser send the "If-None-Match" back again in the next request? It is only done by a standard web request, some browsers disable caching if you press the reload button, some disable it when you press SHIFT-Reload. Try to locate if you have server problem or a browser problem. Use different devices, use different browsers.
- Chrome and Firefox accept the ETag only, if you send a HTML/1.1 header.
If you are using HTML/1.0 - the ETag logic could fail.
Remember: When you send data as HTML/1.1 - Firefox expects that you send the header field Content-Length also! - Is your browser cache disabled? Some browsers have a checkbox in the debugging tools to disable caching.
- Is your browser cache available? I use several portable installations of
Firefox. These installations might have a cache size of 0 bytes. Therefore
the caching is disabled and the "If-None-Match" wouldn't be sent
by the browser.
On Firefox enter about:config in your browser and check the value for
browser.cache.disk.capacity - it must not be zero!
You can guess, why I know that - it cost me one day to find the reason for the missing header field in Firefox. - Add Serial.print debug messages to your code to verify if you receive the HTTP headers fields and print the content to the serial monitor.
Summary
Caching relieves network traffic and shortens the time needed to display an Arduino webpage. Whereof Cache-Control is a simple "one-liner", ETag gives you full advantage on caching. The combination of both methods is possible.