Navigation
Home
Home

Minetest: concealed.world:30000

Nixx's Blog
The Internet is Serious Business.
1 2 3 4 > Next

Automating Writing HTML

What kind of psychopath goes as far as to automate writing plain HTML? Well, me apparently.

At first I didn't think that going this far for automation would be worth it. It was actually my friend over at Saltorn that had suggested the idea to me, seeing as he can automate such things with Emacs when writing things of his own. I thought it would be too far to go in the name of automation, and that the returns would be diminishing.

But in actuality taking out more of the manual effort in writing allows a lot more free flow of consciousness, I tend to find I write more quickly and easily now.

You can find the script here. Although it's suited to my particular usage, you could quite easily adapt it yourself if you're looking to do something similar. I realised that the components I use in my articles could easily be boiled down to a dozen or so different elements, which I could write quick shorthand for.

The elements start consistently with a '@', which is a character I would rarely use, and end consistently with a ';'. Using non-frequent characters means it's less likely there will ever be a case in which it breaks.

And yes, I did write this article using the script, too. It's nice.

2020.11.08 16:00 GMTSee Full Article...

Making a Mess of an E-mail Host

I made an attempt at making my own e-mail server. I was talked into it by Qorg, who reminded me of the existence of the original emailwiz script by Luke Smith. Seeing as I run an Arch Linux server, rather than a Debian-derivative, I would attempt to port it.

I intended to finish this completely, if only for the sake of credibility when I share my findings regarding the process of setting up an e-mail server, but in the end I have left it 98% finished. For various reasons I'll talk about, it wasn't quite worth finishing for me. But still, nonetheless, I hope what I have learned from it can be of use to someone.

How would this in theory be done?

I have made a port of the script, to Arch Linux, which you can find here. If you use a Debian-derivative you can make use of the original script, but for the other handful of weirdos out there that use Arch as a server OS, this may help you. I've taken the time to test most aspects of the script and iron out bugs caused by the peculiarities between Debian and Arch. The README details much of the prerequisites, and the process involved in set-up.

As I have said, I got 98% of the way there before throwing in the towel, not 100%. I am confident that this script should get you most of the way there. It will not get you all of the way there, there will be debugging to be done, but it should make your set-up process substantially simpler than it would have been otherwise.

Is this a good idea to try?

Eh, maybe. Here's some criteria that may help you make up your mind on whether or not you should try it yourself, if you are interested.

You must be able to set up a PTR record (reverse DNS) to your e-mail host. You may need to negotiate it with your ISP, or your server provider. This also typically means you have a static IP. I suspect this was the stumbling block that made my set-up much more difficult, and without it your e-mail server will be much less useful (large providers will filter you).

You should meet more than half of these criteria:

Debugging is your friend

You will in all likelihood run into issues. Here's some good things to try:

Thoughts on the pros and cons of self-hosting mail

Privacy

Naturally, self-hosting will always be better from a privacy standpoint. The number of components you need to trust is smaller. Any other mail provider will be able to read your e-mails if they want to. Even if they offer server-side encryption in the event of a possible attack, this doesn't mean they can't decrypt it themselves.

Client-side encryption will to a degree somewhat level the playing field here. With client-side encryption, any mail you send is illegible to anyone par the sender and the recipient, provided your keys do not become compromised, no matter what e-mail provider you are using. Of course, if your e-mail provider has things like your real name, your phone number, and so on as part of registration, you are screwed anyway.

Security

This one will probably go to larger e-mail providers, rather than to small individual projects. Unless you really think you know security and administration better than teams of people serving thousands - if you do, I have no idea how you got here and why you're reading this. In most cases you will need to thoroughly educate yourself to be able to utilise a comparable level of security - although, it can be argued that a mail client serving a single individual or a few people is also a smaller and less enticing target for attacks.

Reliability

This could go either way. If you can guarantee the physical safety of your server, the lack of network traffic, you may find it to be more reliable and faster than a large provider. Or you may not. It really depends on your server, which you should know better than anyone.

Why have I left the e-mail server unfinished?

Mainly because it would not be worth it to me. As I mentioned, I cannot get a PTR record as I do not have a static IP address, so no reverse DNS lookups. This would make my e-mail less useful, as the e-mail I would send to large providers would be silently dropped and never reach their destination. I planned to finish this largely for the credibility in what I'm talking about. But, I decided to cut this short, because I'm beginning to build up a backlog of other things I'd like to do and things I would prefer to talk about.

My thoughts on some e-mail providers

If you've read all of the above, and would prefer to find a better, more privacy-respecting provider, but without going as far as making your own server, I have experience with some.

Cock.li

Their privacy policy is not bad, you can see for yourself. You can use your own clients. You can sign up via Tor (Cock.li additionally has a Tor address), there are no Captchas last I checked, no personal information is required, and there are no mandatory fees.

Note however that Cock.li is fairly honest about the fact that it's entirely possible to read your e-mails. They are transparent about this on their home page:

Cock.li doesn't parse your E-mail to provide you with targeted ads, nor does cock.li read E-mail contents unless it's for a legal court order. However, it is 100% possible for me to read E-mail, and IMAP/SMTP doesn't provide user-side/client-side encryption, so you're just going to have to take my word for it. Any encryption implementation would still technically allow me to read E-mail, too. This was true for Lavabit as well -- while your E-mail was stored encrypted (only if you were a paid member, which most people forget), E-mail could still technically be intercepted while being received / sent (SMTP), or while being read by your mail client (IMAP). For privacy, we recommend encrypting your E-mails using PGP using a mail client add-on like Enigmail, or downloading your mail locally with POP and regularly deleting your mail from our server.

With the lack of sign-up requirements and the ability to use your own clients - and when combined with use of client-side encryption - I find Cock.li works well for my use case. It sells itself as a joke service, but there are a few serious-sounding domains. People complain that it goes down, but in my experience that has been a number of times I could count on my hand in the last year, for a few hours at most.

Dismail.de

Dismail supports alternative mail clients, you may sign up with Tor without filling a Captcha, and it requires no personal information. The process of signing up requires you to respond to an automated message over XMPP, which in my experience I was waiting an hour or so.

Their terms of service do not sound ideal, and give the impression there are many ways in which you could be banned. One example is below:

sending of messages with the aim of harming or destroying, violate privacy, infringe the intellectual property, to issue statements offensive, fraudulent, obscene, racist, xenophobic, discriminatory, or any other form of content prohibited by law.

Why not Disroot/RiseUp/other privacy-respecting mail service?

As I said, I have tried many, and I found trying to apply to them impractical. RiseUp is an insider-invite-only service, and Disroot has a large list of potential offences you could fall under. Additionally, when signing up to Disroot, I was told to wait because it was the weekend. I came back during the week, and I was told to wait because it was the weekend. I try to balance maximising my privacy with ensuring actual practicality.

2020.11.07 20:23 GMTSee Full Article...

Fighting with Web Standards and the Scrollbar Fiasco

It has been a while. About a month, precisely, although you can see that I've still been alive due to the new posts on my music list. I hope you didn't think I had abandoned you yet.

And as I have done before and will certainly do again, I've got a backlog of posts to spam, that will probably come out over the next couple of days, on various subjects. (For example, the fact that I am writing this post in my own crude markup language, rather than plain HTML).

But first and foremost, to the most immediate change, and in reference to the title of this article. I've been having a back-and-forth with Qorg in regards to this - I like my website without a scrollbar.

A while back it was brought to my attention that this doesn't work with Pale Moon browser - it isn't entirely up-to-date with its CSS seemingly, and if I hide the scrollbar, it is impossible to scroll. I initially thought I had resolved this, although the fix was rather crude, relying purely on the user-agent.

Eventually I needed to create a solution that would work for those that use Pale Moon, but understandably (for privacy, and to avoid dealing those such as Cloudflare) spoof their user-agent. They would not be able to browse my site. And any browsers I did not know of that had the same issue, would also not be able to browse my site. And what if you just preferred to browse with the scrollbar?

So, I have now created the option to browse with a scrollbar, dependant on a single first-party cookie. See the change here. You can visit this page begin using this site with a scrollbar. I personally don't prefer it, but it's not a bad thing for people to have the choice.

That's all for now, more soon.

2020.11.06 21:20 GMTSee Full Article...

My Personal Music List

So, I listen to a lot of music generally, I have opinions on it, I keep a pretty sizable collection. And I thought to myself, "Hey, why don't I make a post for it? It would make for some good content - if I didn't all that knowledge just sits there."

How wrong I was.

It has easily become the largest plain HTML file on this entire site by far. Vim screams in agony. I didn't quite realise I had a problem until I decided to type it all down - and then realise how long that could take me.

Some Statistics (at the time of writing this)


You get the picture. Once I had started it was far too late to stop, my fate was sealed. I do hope however, it is of some value to you. Go check it out:

The music list that haunts the fringes of your nightmares.

2020.09.29 01:35 GMTSee Full Article...

A Therapeutic Mini-Rant About Cloudflare

Normally, I try to avoid reiterating things others have already done. It's not worthwhile to me to get attention by echoing things other people have already said, especially if I haven't learned about it any further through my own experience.

But the reason I come here with this today is that I have even noticed fairly small web projects from independent developers using Cloudflare. It's not even big corporations any more. And I'm not strictly sure people know much about the potential pitfalls, in regards to internet privacy and centralisation. I certainly hadn't read up much on it until recently - all I knew was that Cloudflare left a bad taste in my mouth.

To sum up the basics:

Sources on the privacy of Cloudflare, and how it works on the back-end, aren't terribly accessible. I encourage you to do your own research. But for a starting point that somewhat reflects how I feel about it, take this.

Cloudflare is a piece of software that operates firewall rules, classically to protect your host from threats as such DDoS attacks. There are other options to do this - and even better, you can simply learn how to use iptables yourself.

In short: if your website requires me to connect to Cloudflare and complete a captcha, I'm not going to visit it. If you're already using a website with Cloudflare, seriously consider your use case and whether it is absolutely necessary for you. I would hate to see people acting to centralise the internet without giving it much of a second thought.

2020.09.27 13:42 GMTSee Full Article...

YouTube Ecosystem Wholly in a Terminal

There's a reason for which I sudden got really into RSS as a format. This all started when quite recently, the 'primary' web mirror for the software Invidious, was going down - that being, invidio.us (now hosts a list of other instances). Of course, this didn't mean the other instances would be going down, so I thought little of it. I preferred to use Invidious rather than the normal YouTube front-end, for various reasons related to internet privacy:

Alongside the fact that the default YouTube front-end is an unnecessarily bloated CPU-eater. I believe they have removed the legacy front-end now, though I've not checked in a long time.

Invidious was convenient, as rather than use a Google account for subscriptions - it is effectively just a GUI front-end for an RSS feed. It can also play videos without the need for JavaScript.

But Invidious has its own problems. Hosting tends to be highly unreliable. I can't tell how how easy or difficult it is to host an Invidious instance as I've never done it, but particularly in offshoot instances, they tend to break often and are generally unreliable, and sometimes inaccessible.

Returning to the Original Story

So, before long, invidio.us went 'down', as expected. But I noticed something strange begin to happen. As I used other instances instead, it seemed like they were breaking even more often. If that were true, it would be highly conspicuous - it would imply that in some way, these offshoot instances depended on the primary instance, and were now running into problems. That's quite troublesome design. Of course, I can't statistically prove that is true without taking a serious look at the codebase, which I don't plan to - it certainly could just be cognitive bias. But it got the ball rolling to make me begin to consider my alternatives.

YouTube offers an RSS feed to channels by default. You can in fact subscribe to YouTube channels in an RSS feed reader - this is exactly how Invidious works. There are RSS feed readers - such as newsboat - where you can read RSS feeds from a terminal. Put 2 and 2 together - I can see my YouTube subscriptions from the terminal.

So I went ahead with that

It all worked perfectly well. I added a few macros, which you can run via ",<key>" in newsboat, to do things such as - play audio, play video, download audio, download video, and open in Qutebrowser. You can see what I've done below:

macro v set browser "mpv --ytdl-format=22\/18 --ytdl %u" ; open-in-browser ; set browser "lynx %u"
macro V set browser "youtube-dl --add-metadata -f 22/18 %u" ; open-in-browser ; set browser "lynx %u"
macro a set browser "mpv --ytdl-format bestaudio --ytdl %u" ; open-in-browser ; set browser "lynx %u"
macro A set browser "youtube-dl --add-metadata -x --audio-format mp3 %u" ; open-in-browser ; set browser "lynx %u"
macro q set browser "qutebrowser %u" ; open-in-browser ; set browser "lynx %u"

This was great. And then I wanted to search for a song I didn't have a copy of, and didn't quite remember the name of. And then I thought, "Wait, how am I supposed to search?"

A TUI client for YouTube

The first thing I did, rather than reinvent the wheel, was to look to see what else others had done. But it was all pretty poor.

Generally they were written in Python. Almost all of the time they didn't work. The few that did 'work' packaged a bunch of other things I didn't need - like a video player, when I already had mpv. I just needed the ability to query.

And so, I wrote my own

The program, 'ytsearch', offers you this functionality in selecting queried videos:

Data showed in search queries:

Actions on selected videos:

Preferences which can be set as parameters or in a config file:

Beyond a standard GNU/Linux system with coreutils - this program depends on mpv, youtube-dl, and to a minor extent on xclip (which makes video sharing more convenient - no functionality is broken by editing it out).

At first, I tried to use the functionality packaged with youtube-dl - 'ytsearch' - which can be used to get one result, 5 (ytsearch5), or however many you can (ytsearchall). But I found this to be an extremely slow process, that was difficult to format and work with. I also think it is 'unfriendly' to YouTube, as it makes many video requests in formatting a single list of search results. This led to me getting banned very quickly.

So I pretty quickly ditched that idea. I couldn't curl YouTube either though - their results don't appear without JavaScript enabled. So, I settled for curling Invidious instances for links (their format is pretty universal across instances, I've not noticed any problems yet), and then passing those links to download/stream from YouTube itself. This method is much 'friendlier', and by far much faster to query - even for playlists with hundreds of videos (which are displayed in pages you can scroll through).

Drawbacks and Concessions

It has some drawbacks, of course. It is very scrappy scripting - this program evolved as I needed it to - and as such the code is an absolute shambles. It at least does work. Naturally, it's not even close to POSIX compliant, and depends heavily on Bash. I could likely have made a better program if I bothered learning ncurses, but I don't use YouTube often enough to justify that kind of effort.

It also only works as well as your Invidious instance works - if the instance has problems, so will the script. To mitigate this, I've made it very easy to test and change instances via parameters and config files. It reports reasonably well on what the issue could be if it fails to fetch videos.

Companion Programs

'ytsearch' has two companion programs - 'ytformat' and 'ytchannel'.

'ytformat' just takes links pointing to an Invidious instance, and makes them a YouTube link, if you are looking to share them as such. Fairly straightforward.

'ytchannel' is a fairly hacky script, that downloads and deletes a low-quality audio copy of a video, in order to get the YouTube channel it belongs to and to give you it in RSS feed compatible format. Thus, with the 3 programs, it is entirely possible to run the full YouTube 'ecosystem' from a terminal, and to never need to leave a terminal, and still have advanced search features and channel subscriptions.

Getting the Scripts

You can find all of the aforementioned scripts on my Git server, in a repository of their own, here. I hope they work for you. I'd also love to see - if anyone is interested in it - improvements to it, and probably written in something better than Bash.

2020.09.25 14:39 GMTSee Full Article...
1 2 3 4 > Next

BTC: 18XfSy6o4uozbFdWN2ixTy1XZjt5jKxwkg

Contact Me: nixx@firemail.cc