Improving our internal RAG, AskNow, as we use it

SYSLABO is using and improving AskNow, an internal RAG system that answers questions based on internal information.

By checking answers while using it ourselves and organizing the information it refers to, we are bringing its answers closer to what helps in daily work.

Our aim is to reduce the time spent searching for necessary information and checking its content, so that members can move on more easily to their next decision or action. To do this, we are reviewing both how the AI answers and the internal information its answers are based on.

• Having information does not mean it can be used right away

  When looking something up inside the company, finding a document does not always solve the problem.

  Is the information up to date? Does it apply to my situation? Are there exceptions or notes in another document? Using information in actual work may require checks like these.

  Sometimes people cannot decide even after reading a document and have to ask a member who knows the background. The person asking waits for an answer, and the person answering stops their own work to check, so confirming a single piece of information can take up the time of several people.

  To reduce these situations, it is important not only to gather information in one place, but also to make clear what content is needed and the conditions for using that information.

  With AskNow, we aim for an environment where members can move their work forward while checking information, by connecting internal knowledge to answers to everyday questions.

• Referring to internal information to create answers that fit the question

  RAG is a mechanism in which information relevant to a question is retrieved from documents and other sources, and generative AI creates an answer while referring to that content. It is used as one way to handle company specific information that general knowledge alone cannot answer.

  However, preparing a mechanism that refers to information does not mean that every question can be answered properly.

  If the necessary information is not included in the documents, or the part that matches the question cannot be found, the answer will also be lacking. Even when the referenced content is correct, if conditions of application or points of caution are left out of the answer, users may be unsure how to decide.

  For this reason, in improving AskNow, we focus not only on whether the text is easy to read, but also on whether it includes what the question needs and whether it matches the information it is based on.

• When reviewing answers, we also check the information behind them

  When we do not get the answer we expected, the AI’s wording is not always the only thing to fix.

  For example, even for an answer that lacks explanation, what needs to be reviewed differs between a case where the original document does not contain the necessary information and a case where the document contains it but the answer does not reflect it.

  In the former case, the information must first be added. In the latter, we need to check how information related to the question is searched for and how the referenced content is conveyed.

  When documents contradict each other, a person must also confirm which content should be the standard.

  The review of answers and organization of reference data that we are carrying out is an effort to consider these problems together. We believe that continuous improvement comes from checking the causes of users’ confusion, rather than stopping at polishing the surface of answers.

• Organizing information means sharing the premises of the work

  Organizing internal information means more than making text easier to read or sorting documents.

  For example, even if a document describes how to handle something, users cannot decide unless they know whom it applies to, when it takes effect, and in what cases a different response is needed.

  Some premises seem obvious to members who have handled the work for a long time but are unknown to someone taking it on for the first time. Explanations that used to be supplied by asking someone may not be left in documents.

  Using RAG in business requires paying attention to these premises and conditions as well.

  Existing ways of operating have reasons why they were chosen and ideas built up in the field. When organizing information, we want to understand that background and share it in a form other members can use.

  Questions asked to AskNow also serve as clues for thinking about where explanations need to be added within internal knowledge.

• Finding points to improve from actual questions

  Users do not always ask questions using the formal names written in documents.

  They ask what they want to know by shortening the names of tasks or explaining the situation they are in. Even when people need the same information, the words and level of detail in their questions vary with their work and experience.

  That is why the questions that arise in actual use contain clues for improvement that developers alone may find hard to notice.

  If someone asks the same point again after reading an answer, the explanation may be lacking. If the information is found but the next action is unclear, the conditions or contact points needed for a decision may be missing.

  With AskNow, we treat everyday questions and answers as opportunities to review answer content and reference data.

  After making improvements, it is important to check whether the questions that originally caused trouble can now be answered, and whether people can reach the necessary information with different ways of asking. By repeating use and review, we bring the system closer to something easy to use in daily work.

• Handling cases where the evidence is not enough

  AI used in business also needs to make clear what it can answer and what needs to be checked.

  If it returns a plausible answer when the available information is not enough to decide, users may move their work forward on a wrong premise.

  Is information lacking, do the conditions of the question need to be confirmed further, or is a decision by the responsible department needed? If that difference is clear, users can more easily think about what to check next.

  In improving AskNow as well, we do not aim only to increase the number of answers; we value being able to use it with confidence while checking the evidence.

• Connecting internal knowledge to the next piece of work

  As there are more situations in which people can check the necessary information themselves, those asking can spend less time waiting for answers. Those with knowledge can also reduce the burden of giving the same explanation repeatedly, and more easily spend time on consultations that require individual judgment and on new challenges.

  This requires continuously reviewing both the AI’s answers and the information shared within the company.

  While using AskNow ourselves, SYSLABO is confirming what makes an answer useful and what information is needed for it.

  We will continue to use and improve AskNow so that the knowledge and experience each person has can also support the decisions and actions of other members.