Fresh Takes & Updates

As the highly anticipated engineers' conference rapidly approaches on the 24th of this month, our team is buzzing with excitement and dedicated efforts, steadily moving forward to unveil our groundbreaking AI-powered assistant, SuperPower.Engineer.

This blog post gives an in-depth look into our recent developments, showcasing the significant innovations, technical advancements, and enhancements we've achieved . . .

Bringing Ideas to Life: Interactive Dummy UI

To give life to our vision, we developed an engaging and interactive dummy  user interface using Streamlit. This rapid prototyping strategy enabled our team to experiment extensively, iteratively refining and optimising the interface.

Through continuous testing and feedback, we've created a seamless and intuitive user experience, closely aligning with real-world user expectations.

Our dummy / prototype UI has proven instrumental in identifying potential usability issues early, allowing us to enhance user interaction flows significantly before the final implementation.

Meme Magic: AI with a Sense of Humour

Injecting personality and creativity into our AI assistant, we introduced an engaging "Meme Mode," comprising 20 thoughtfully crafted, humorous prompts. This feature showcases Superpower’s playful side, adding an engaging dimension to user interactions.

The addition of Meme Mode underscores our commitment to providing a multifaceted AI experience that resonates with users, making interactions memorable, enjoyable, and relatable. This element of fun helps establish deeper user connections, fostering loyalty and satisfaction.

Deep Web Diving: Advanced Web Scraper

Understanding that a comprehensive knowledge base is fundamental to AI accuracy, our team developed a sophisticated web scraper capable of navigating through multiple layers of web pages.

This advanced scraper effectively captures detailed HTML content and extracts embedded PDFs, substantially enriching our assistant's knowledge repository.

The depth and breadth of data gathered enable Superpower to deliver precise, well-rounded, and highly informed responses, significantly improving its practical utility for users across diverse contexts.

Memory Mastery: DynamoDB Integration

Recognising the vital role of context in conversational AI, we implemented an advanced memory management system utilizing DynamoDB. This robust solution reliably tracks usernames, user IDs, conversation IDs, and chat session data, enhancing the continuity and personalization of user interactions.

By meticulously integrating user IDs and comprehensive session tracking, we've ensured that returning users experience seamless conversations, maintaining relevant context across multiple interactions.

This ensures a personalised, intuitive, and cohesive experience each time a user engages with our AI.

Seamless Session Integration: User ID and Chat Sessions

To further enhance personalisation and continuity, we've seamlessly integrated user ID and chat session tracking. This integration allows Superpower to recall previous interactions effortlessly, delivering context-aware, relevant, and personalised conversations.

Such an intuitive approach significantly improves user satisfaction, creating a consistently engaging and meaningful conversational experience. It also enables robust analytics, helping our team refine and continuously improve the AI’s performance and user interactions.

With the demo day fast approaching, excitement within our team continues to soar. Each accomplishment and innovation brings us one step closer to presenting an exceptional demonstration of Superpower's cutting-edge capabilities.

We’re confident our AI assistant will leave a lasting impression, showcasing the profound potential of technology-driven solutions. Stay tuned as we finalise our preparations—an extraordinary experience awaits!

Integrating Memory into our LLM

Over the past two weeks, we've significantly upgraded our Retrieval-Augmented Generation (RAG) pipeline by integrating memory, ensuring our model remembers and utilizes previous interactions to enrich future responses.

By nature, Large Language Models (LLMs) are stateless. Each interaction stands alone without memory of previous conversations. To overcome this limitation, we've integrated a powerful framework offering multiple memory strategies:

• Conversation Buffer Memory: Retains the entire conversation history, providing full context in subsequent interactions.

• Conversation Buffer Window: Stores only a specific number of recent interactions, managing memory size efficiently.

• Conversation Summary: Generates summaries of past conversations, allowing the model to retain context without overwhelming it with details.

For our specific use case, we opted for Conversation Buffer Memory, specifically retaining the last five conversation exchanges, balancing detailed memory with performance efficiency.

Pinecone Integration and Multi-user Management

As we refined our RAG pipeline, we incorporated Pinecone, a test database, to systematically store conversation histories. Each interaction (user query and AI response) is indexed in Pinecone, effectively serving as a persistent memory store.

We advanced our setup further by assigning unique index within Pinecone for different users. Each user's conversation strings are independently stored within the user-allotted index under a specific namespace, which enhances user data management and security.

Tackling Cross-contamination in Document Retrieval

A critical issue we addressed was the potential cross-contamination, where the LLM could inadvertently mix documents from unrelated contexts to generate answers. Our solutions involved:

1. User-Specific Concatenation: Identifying the user's specific system type (e.g. type of alarm system) and appending it directly to their queries. This ensured retrieval of only relevant documents tailored to the user's context.

2. Namespace Segregation by Manufacturer: Creating separate namespaces within Pinecone for each alarm system manufacturer, further refining document retrieval and eliminating cross-contamination.

We're continuing extensive research to further optimize this aspect in the coming weeks.

Ninja Mode: Advanced Reasoning Activated

We introduced an exciting new feature, named Ninja Mode, transforming our AI assistant into an advanced troubleshooting expert. When activated, Ninja Mode leverages OpenAI’s state-of-the-art reasoning model, o1.

This advanced model enables the assistant to intelligently gather comprehensive contextual information, analyse complex situations methodically, and deliver highly precise troubleshooting insights.

Ninja Mode essentially provides our users with an expert-level AI technician capable of handling intricate queries with advanced reasoning capabilities.

Stay tuned as we continue to push the boundaries of AI capabilities, optimising our RAG pipeline further and exploring new frontiers in memory management and advanced AI troubleshooting!

Out of the gate we've primarily focused our resources on working out how to retrieve the most accurate answers possible from complex instruction manuals (our core product goal), by extracting & matching chunks of both text and image information - including the ability to extract an understanding of content embedded in images (which is - we found - a pretty complex challenge). 

Key progress to date includes:

• Document Processing and RAG Pipeline Development:
  

  • Focused on refining document processing capabilities for a Retrieval-Augmented Generation (RAG) pipeline.

  • Conducted research and integrated a data structuring system to improve document parsing and processing - supporting data inputs from Amazon S3, with data storage in Pinecone vector database.

  • Enhanced data parsing by enabling:

    • Image extraction for detailed content descriptions.

    • Table extraction for structured data representation.

  • Implemented text chunking to improve similarity matching in the vector database.

  • Utilized OpenAI’s text-embedding-3-large model for vector embedding.

• RAG Pipeline Creation:
  

  • Built and tested a RAG model and pipeline using Pinecone’s API and GPT-4o for answer generation.

  • Initial results demonstrated the pipeline's strong performance.

Following a review of the initial results we identified we were consistently generating accurate answers from in-depth complex manuals (which was the good news), however we needed to include wider contextual information to enrich and support the answers (e.g. including citations and more accurate / contextual image information).

Features & testing that followed included:

• Citation Integration:
  

  • Incorporated source citations directly into LLM inputs to embed accurate citations within generated responses.

  • Enhanced the transparency, credibility, and traceability of the responses.

• Detailed Source Referencing:
  

  • Extracted detailed metadata from Pinecone, including document IDs and specific page numbers.

  • Organized related documentation into an Amazon S3 bucket for easy access to original source PDFs.

  • Enabled users to verify facts by referencing the exact source pages.

• Visual Guidance via Images:
  

  • Attempted to display images from referenced pages but faced limitations with metadata and base64 encoding.

  • Acknowledged the need for improved visual aids for complex questions (e.g., wiring diagrams) and planned for future refinement of this feature.

• Incorporation of Internal Knowledge:
  

  • Integrated a structured dataset from a CSV file into the pipeline, including specific private knowledge sets like:

    • Terminology

    • Expert internal troubleshooting knowledge 

  • Improved the quality of generated responses by enriching them with highly accurate, contextually relevant internal knowledge.

• Natural follow-up questions:

  • Added x3 follow-up questions per query, to dig deeper into the solution, for example, when programming is described at a top-level, the option to get the step-by-step programming information is provided.

Having made good progress retrieving accurate answers and providing citations (text and image) to help contextualise the answer, we're now moving onto dealing with more 'complex queries' - i.e. system-based queries that may involve more than a single 'instruction manual answer call', and involve various elements of more complex troubleshooting.

Hang tight for more progress and the next update - it's starting to get interesting ;)

Abdur spent a day out on the road (in Easingwold, rural North Yorkshire) with Engineer Phil to help understand more first-hand the kinds of problems field-service engineers face out in the field. Observations from the day were:

• YouTube: Engineer Phil's current No.1 trouble-shooting tool is YouTube. He said that he found the visual nature of it much more helpful than Google / instruction manuals, and referenced the ability to play / stop videos, to see how to fix an issue.

• When trouble-shooting, Phil uses a mix of:

  • Experience / knowledge

  • Instruction manuals

  • Senior tech support (via phone)

• Seemingly in most cases, if the instructions / problem-solve was available easily 'on tap' / 'on site' / 'on phone' Phil wouldn't need to speak to Tech Support (e.g. when Phil called colleague Asif to help with job 1, Asif's solution was to send an instructional diagram showing 'which wires / pins should be connected' to Phil - and Phil then followed this to solve his issue)

• Phil highlighted that 'visual diagrams' / imagery is highly useful in the field (hence his love of YouTube)

• For Abdur and Phil's 2nd customer visit, they encountered at 25 year-old vintage alarm system (the Paragon Super 2) This type alarm system is something that Phil had no specific experience of, so he immediately searched out the instruction manual on Google to find the answer. As a challenge, Abdur also used ChatGPT to find the answer - and ChatGPT proved a lot quicker in finding the answer, mostly as although Phil found the instruction manual - he then needed to trawl through multiple pages to try to find the right page - whereas ChatGPT served up an answer quickly on a plate.

Other insights from the day include:

• Engineer Phil had never used ChatGPT before, and when we showed him it (i.e. fed a technical question to it, and then saw it spit out an answer) his jaw dropped. It was a magic moment.

• Experience is seemingly as important as 'instructions from manuals' - as often things don't work as they should.

• Methodical trouble-shooting is definitely a skill - but also definitely something that can be taught / explained (mechanically). Phil talked through a series of examples of how he has tackled problems in the past.

This morning, we had a local plumber come to fix a few random plumbing issues we've got around the house. The first problem we asked Plumber Matt to look at was our basement water pump alarm, which had been randomly beeping for a while, and we weren't quite sure what the problem was.

Enter Plumber Matt, who did the following:

• Checked the pump alarm make - 'Edincare AquaSafe'

• Told us he didn't know anything about our specific alarm

• Went and Googled the make to see if he could find anything about it

• Google came back with a generic answer, so he gave up

• So told us he couldn't fix it, and referred us to another specialist £150 a visit company, to fix it

What Plumber Matt should have done is what I did, which was:

• I went onto AskAMCO (our custom GPT - and SuperPower v1 Prototype)

• Entered the prompt 'Edincare AquaSafe alarm beeping and flashing, how to fix?'

• Within 5 seconds, the machine told me to press a couple of buttons in various sequences, and hey presto - like magic, the problem was fixed.

Screenshot

Lessons learnt?

• If only Plumber Matt knew about Superpower.Engineer and the power of AI

• Google is comparatively scarily bad - even in comparison to our V1 (which is itself about 80% good)

• I just saved myself £150

In the context of building a tool designed to share knowledge using Artificial Intelligence there's one fundamental question that's been bugging me, which is: 'how can a computer teach itself' - or to put it in a more grounded way: 'how does a computer know that 2+2=4 without being told / taught so by a third party (i.e. a human)?

Of course, the concept of knowledge and learning is no different when talking about machines as it is with humans, so the basic question is 'how do we learn stuff'? (and what is it to 'know' something?).

Listening to a podcast with Ilya Sutskever (one of the founders of OpenAI) he described very succinctly that humans learn in two ways:

• Learning from others

• Learning by trial and error (& patterns)

So in the context of a computer not being able to 'learn from someone else' then the only other way it can learn is by trial and error - or a more smarter version of trial and error being 'pattern matching'. For example, the computer may observe that 1+1 = 2, and therefore guess that the pattern follows that 2+2=4.

It's this 'guessing' that underpins the intelligence and knowledge of AI. Essentially all AI is doing is a very sophisticated very of guessing (aka probability) - and thanks to extremely large neural networks powering Large Language Models (LLMs) this 'guessing' is next-level guessing (aka highly technical and increasingly accurate probability modelling).

Other than listening to AI OG's like Ilya Sutskever for insights into how humans and machines learn, I've also started quizzing my 8 year-old daughter for insights into learning - as she is a teaching obsessive and has a healthy love of writing school lesson plans in her spare time.

Interestingly, her answer to the 'how do we know 2+2 = 4' question was bang on Sutskever's answer - telling me: 'Well Daddy, it's basic maths if 1+1 = 2 then of course 2+2 = 4'.

The simple pattern matching of an 8 year old's brain underlines a fundamental truth of AI - that all intelligence is essentially 'guessing' (or a very accurate version of guessing based on probabilistic pattern-matching).

As a core goal of SuperPower.Engineer is to prove the transformative power of AI on the physical work environment, we also thought it would be fun (and useful / efficient) to shape the wider project using as much AI tech as possible - starting with the Logo.

As of today, AI is 80 to 99% good at a lot of tasks. Logo (image) generation I'd say is around the 85% mark.

Here's the journey we took in creating the SuperPower logo - using a mix of ChatGPT and human design talent.

Step 1: ChatGPT prompt 1

The first thing we did was feed a prompt into ChatGPT along the lines 'Make us a logo for our new AI start-up called SuperPower. . . '. (You can read the full prompt & process via ChatGPT here: https://chatgpt.com/share/67b5f6f1-a0e4-800b-85f3-2d7a6ee8247d)

Within about 60 seconds - after a bit of questioning re: some design specifics - ChatGPT delivered an initial design, which looked like this (which I'd say was about 20% good):

Step 2: ChatGPT prompt 2

Mildly unhappy with the output, I then gave ChatGPT a further prompt to include 'SP' in the middle of the logo (which was unclear in the original), along with the word 'SuperPower', and miraculously it spat out the following - which I'd say was about 85% good:

Step 3: Dave the designer

To get it to 100% good, I decided to ditch ChatGPT and pass over the v1 design to my erstwhile designer friend Dave - who then took another c.30 mins to tweak it & turn it into what we have now.

Here's the final stage brief I have to Dave - using the core initial design from ChatGPT:

And here's the final design output then cleaned-up and created by human designer Dave:

The whole process (from a my time perspective) realistically took about 15 mins (which was basically my briefing up Dave with the basics that ChatGPT created), and another 30 mins from Dave - meaning end-to-end, our logo took about 45 human minutes to create. With a rough 60% AI and 40% human input ratio.

Where better a place to build an AI App than from the British Library - home of the Turing Institute (the OG of AI) and neighbours to the Crick Institute (the neurobiology powerhouse).

SuperPower HQ - the Rare Books and Music room of the British Library

The Alan Turing Institute (our neighbours)

The Enigma Machine

Come and say hi if you're close by ;)