General programming discussion • Re: 'Best' language for user Services

Assuming you don't buy an rpi with more ram..

A valid assumption! I do have a 4 GB Pi 4, which I use for development, but my home control stuff has been running (with occasional added functionalities) on my 2B for years, and the problems with declining memory have only recently surfaced. Incidentally, increasing my swap space seems not only to have restored some free swap space in the short term, but it also seems to have (at least) greatly reduced the tendency for free space to decline in the somewhat longer term. I have needed to reboot the Pi rather more often for other reasons over the past couple of weeks, so I am not 100% sure that this is a true cure.

I suspect if you (say) moved the sqlite3 files onto your NAS with a mountpoint to your rpi the same thing will happen. It might not but that would be my expectation. If you're familiar with setting up NFS then it would be a quick test otherwise not worth the effort.

That didn't seem particularly worthwhile to me.

If you were to change from sqlite3 to a remote database (eg: mariadb) then I'd expect less buff/cache to be used because it would be going over the network. Obviously that would require code changes plus a remote db to talk to: substantial effort if you've not used "mysql" before.

I actually have quite a lot of experience with MySQL/Mariadb for web development on larger systems, so I am in the process of moving my databases to Mariadb on the NAS. The one disadvantage of this is that the Mariadb connector on the Pi is pip installed, so that I will also have to run all the services in a virtual environment. I am already doing this with a couple, so it is a nuisance rather than a major problem.

I don't follow why a pip installed mariadb connector requires a VM?

I have a pair of (ancient) spinning rust x86 HP Proliant NAS. They only do samba/nfs/iscsi. I'd never install something like mariadb directly onto them. If forced, I'd have one of the NAS run a VM in which the mariadb existed. Is that what you meant?

I've not tried creating a VM using an rpi although that is the direction I'm heading.

Does sqlite3 have the notion of a db "transaction"? Basically you tell your db server every operation between "transaction start" and "transaction end" is an atomic operation. ie: it all succeeds (committed) or it all fails. It can be considered the db equivalent of a mutex.

Yes, it does, though it is not a feature that I use - I don't see any point in an environment when only one service is writing to any Table.

Apologies. I didn't intend to suggest only one client can write. Multiple clients can write. I'm out of my comfort zone explaining further because methinks it rather depends on the DB implementation and I'm not a DBA. The idea is it prevents two clients writing to the same thing at once. Until the transaction completes the other clients get queued. The definition of "thing" is what I don't know. A naive DB might lock the entire table(s) involved! I'd expect something more intelligent these days but with sqlite3 being "file" based and typically expecting a single client who knows?

It doesn't actually matter from the paging aspect. My thought wrt to a transaction was it has to be committed which (in principle) would mean the OS can release that memory and thus it might help paging. It also follows that memory might immediately get gobbled up by a client trying to read but then again, if your apps are all reading the same thing then the OS only need to hold one copy. Very tricky to get to grips with.

Have I mentioned zram? I think I held off on that. If the extra page space turns out to be of help then zram might nudge the rpi over the fence into acceptability. It comes at the cost of extra cpu cycles because it's compressed paging. There are folk on this forum who have said zram works much better than one would expect. We can imagine most of your page file will be full of text. That compresses well (and fast). Off the top of my head you might care to run 'hexdump -Cv" on your pagefile to see if most of it is text.

I'd suggest looking into why things won't stop/start without manual intervention. The notion of a transaction might solve this. Have the sole "writing" service do it as a transaction. This will have the "reading" services either block or fail (may reveal the bug). It's possible the write transaction might free up buff/cache because it has to be written to disk. Dunno if sqlite3 will cheat on this.

Under all circumstances except rebooting, the services start and stop cleanly. I think that there is just too much else going on during the reboot.

You can have systemd bring up (and take down) services in the right order. I haven't bothered to learn much systemd. I tend to have it call one script which does it the old fashioned (/etc/init.d) way by calling other scripts.

I am fairly experienced at writing systemd services. It seems to me that its job ordering doesn't always perfectly obey the instructions and/or its own rules!

You're not the first to say that which is why I've been in no hurry to move with the times.

Statistics: Posted by swampdog — Sun Jun 22, 2025 3:36 am

---