AI Projects and Thoughts

I am going to prefix this with I have a complicated opinion of AI. In 2021 I filled a patent in the area of artificial intelligence with respect to scaling it\integration with consumer data. I also adopted ChatGPT early on and used it as an aid for searching obscure errors in 3rd party products. On a personal level I have made very effective use of it.

I also fundamentally have an issue when it is used without understanding the output in the case of software engineering. I've seen it as a great way for people to shoot themselves in the foot. My biggest fear is concerns over atrophy of ones cognitive abilities.

My stance today is somewhere in the middle. AI is excellent at writing boilerplate, repeating patterns, and doing mundane tasks. That said a lot of this a good library\framework can do for you. It fails at proper architecture and design. Many vibe coded applications today have significant architectural issues, stability issues, scalability issues, and security issues. This in addition to the code being harder to read both because of how it may have been written and due to the engineer having not written it themseleves. All this said...I view AI as a tool and for certain types of development it is a strong enough work accelerant that ignoring it feels like a poor decision from a professional standpoint.

My usecases today

I'm going to explain my usecases today so that the projects I list make more sense.

Searching obscure errors. Effectively if I'm just doing a Google search anyways I might as well let the AI take a look while I search in a different tab, perhaps it will surprise me with a working solution. Often it doesn't, but every so often it saves me a few minutes of searching especially if it's something obvious I am just missing.

Writing boilerplate tasks that have been done dozens of times before. For example when working on Athena I had a complete working CLI before the REST API. REST APIs after you have already implemented the model layer\logic are fairly copy\paste in many cases. Effectively a path, function, etc. In this situation I did say ok...I'm going to let this generate and then I will review the output when I get back in about 20 minutes. It wasn't perfect by any means I did have to rewrite some sections where things clearly were incorrect, I did introduce some security middleware I wrote by hand because I was not taking chances on that, and refactored out duplication. But it saved me on a non-trival amount of time and given similar circumstances I would be ok doing that again.

Writing one-off throw away applications that I am not willing to write by hand due to time. For example I have a gmail integration that summarizes my emails with a local LLM model. This allows me to at a glance be reminded of emails I am already keeping an eye on. But I would never dedicated the time to writing it myself because it's just not worth it. The reality here is if it breaks I'm perfectly fine just letting an AI rewrite it or not use it again.

High level of logs for my servers\applications. Reality is I do not have the time to review them given at one point I had over 30 VMs on my local network. Having an AI summarize these is not the worst idea.

Projects

I have an application I wrote which just sends requested applications to AIs on the network running on local infra (Ollama atm). They make an attempt and if they fail cool. If they manage to write it cool I get a new tool that I wasn't willing to spend time writing. If they fail cool I didn't spend any money beyond the electricity. This works by batching the requests and turning on the respective hardware as needed based upon the task requested typically running during the night. It's effectively a stripped down k8s implementation for offloading jobs to the AI hardware.

My partner and I have an AI agent on our network that is developed by both of us. It has around 400GB of unified memory running in distributed compute on Mac hardware. It's also a memory based agent so there is limited ability to learn. Due to the limits of local hardware I have spent time writing tools to augment the capabilities by allowing lookups of information on the network as well as rewriting multiple tools available to be less error prone\have a faster execution time. This is largely because 1-2 minutes per tool call is normal in that type of setup so if you design tools to only require one or two calls to acomplish a task you can have a significantly faster\less error prone result.