[BUG]ish: Local storage overflow

#1

Storing all of the scrollback will quickly overfill the localstorage, and run into quota problems. The localstorage log may need to be pruned at the head at a certain buffer size. Perhaps configurable?

main.54554643.js:16296 Error writing char log: log.1618161430551.c1piud8t874db56spdb0.bvh05fot874ehfs7rtfg DOMException: Failed to execute ‘setItem’ on ‘Storage’: Setting the value of ‘log.1618161430551.c1piud8t874db56spdb0.bvh05fot874ehfs7rtfg’ exceeded the quota.
at e.value (https://wolfery.com/main.7927cb11.js:1:267206)
at e.value (https://wolfery.com/main.7927cb11.js:1:265445)
at https://wolfery.com/vendors~app.52a210f3.js:1:28112
value @ main.54554643.js:16296
main.54554643.js:16296 Error writing char log: log.1618163184487.c1pjc3ot874db56spddg.bvh05fot874ehfs7rtfg DOMException: Failed to execute ‘setItem’ on ‘Storage’: Setting the value of ‘log.1618163184487.c1pjc3ot874db56spddg.bvh05fot874ehfs7rtfg’ exceeded the quota.
at e.value (https://wolfery.com/main.7927cb11.js:1:267206)
at e.value (https://wolfery.com/main.7927cb11.js:1:265445)
at https://wolfery.com/vendors~app.52a210f3.js:1:28112
value @ main.54554643.js:16296
main.54554643.js:16296 Error writing char log: log.1618171506476.c1pld48t874db56spdjg.bvh05fot874ehfs7rtfg DOMException: Failed to execute ‘setItem’ on ‘Storage’: Setting the value of ‘log.1618171506476.c1pld48t874db56spdjg.bvh05fot874ehfs7rtfg’ exceeded the quota.
at e.value (https://wolfery.com/main.7927cb11.js:1:267206)
at e.value (https://wolfery.com/main.7927cb11.js:1:265445)
at https://wolfery.com/vendors~app.52a210f3.js:1:28112
value @ main.54554643.js:16296
main.54554643.js:28086 Uncaught DOMException: Failed to execute ‘setItem’ on ‘Storage’: Setting the value of ‘lookat.bvh05fot874ehfs7rtfg.bvh05fot874ehfs7rtfg’ exceeded the quota.
at e.value (https://wolfery.com/main.7927cb11.js:1:461109)
at e.value (https://wolfery.com/main.7927cb11.js:1:460436)
at https://wolfery.com/vendors~app.52a210f3.js:1:28112

#2

Great catch, Max! :slight_smile:
I did implement some auto-pruning to be performed when the error occurred, but I never tested it. Clearly it didn’t work as expected.

A quota sounds like a good idea. I will add it ASAP.

Yes

#3

The need for pruning is another incentive to implement live logging to a file, since presumably anything pruned will no longer be available to save to disk manually.

#4

Incidentally, I’ve been the park since April 1 and my scroll back still seems to have all of that text (a total of 4.9 MB) across multiple sessions, the largest of which is 990 KB. How do those figures compare with yours, Maximus? The issue may be with the size of a single log vs. the total size of session data. I have no idea, just guessing.

#5

I would love this too. But this is unfortunately not possible for web apps (afaik), only for native apps. Web apps are limited by browser security:

  • Writing to files is not allowed (eg. streaming logs live)
  • Downloading of files (and thus logs) is allowed on user interaction (eg. a click). This is what we have with the log export function.
  • A web app can write to a local database called localStorage (or sessionStorage). This is what Wolfery uses.
  • localStorage storage space is limited by the browser, often to a total of 5MB. This is the limit that Maximus ran into.

There is no way to get around these hard limitations without resorting to logs being stored on the server, which I won’t do.

So, I don’t think we have much of a choice, other than pruning. I am happy to be wrong though! :slight_smile:

@Lochiel, how do you measure size?
If you are checking the file size of the exported logs, they will most likely be larger in size than the space used in localStorage. Correct would be to calculate it running a bit of Javascript in the browser’s developer tools console.
See this Stackoverflow answer: javascript - How to find the size of localStorage - Stack Overflow

/Accipiter

#6

Oh, it turns out what you see is my pruner in action. The code is:

let retry = 100;
while (retry) {
	try {
		store.setItem(key, data);
		return;
	} catch (e) {
		console.error("Error writing char log: ", key, e);
		// Probably out of space. Try to remove oldest key
		if (!this.keys.length) {
			return;
		}
		retry--;
		if (!retry) throw e;
		store.removeItem(this.keys.shift());
	}
}

Basically, if an exception occurs when trying to write the log (store.setItem), it will be caught and log the error you see. Then it will try to take the older log entry (if one exist) and remove it from the store (store.removeItem). Once removed, it will try again, and repeat like that until it is able to store the entry, or has retried 100 times.

I still think a quota would be better than to act when the store if full. A full store can cause other problems for other modules trying to use localStorage.

#7

Is it not possible to receive a click making a request to write a file, get the user’s permission to perform the write operation, and then simply keep the file open and keep writing to it?

#8

Not that I can see. There is a draft for FileSystemWritableFileStream, which seems to allow it. But nothing that is yet implemented in browsers, afaik.

However, the IndexedDB API could be an option, as it allows me to store larger amounts of data client side. I haven’t used it before, so I will look into it more.