Bringing doctors together

or

A free, private network for verified physicians

Leading voices

Dr. Nneka Ufere

Liver Transplant · MGH

View

Dr. Vinay Jahagirdar

Liver Transplant · Medical College of Virginia

View

Dr. Lauren Feld

Liver Transplant · UMass Memorial

View

Dr. Sonali Paul

Liver Transplant · UCMC

View

Dr. Pierre Gholam

Liver Transplant · CWRU

View

Dr. Elliot Tapper

Liver Transplant · UM

View

Dr. Saurabh Sethi

General GI · Oja Health

View

Dr. Sara Horst

IBD · Vanderbilt

View

Dr. Wendi LeBrett

General GI · ModernGut

View

Dr. Treta Purohit

General GI · Oshi Health

View
  • Discover your community

  • Stay current

  • A profile that represents you

  • Find your people

About RoonPrivacyTerms

    Overview

    Hepatology e-consult responses generated by artificial intelligence demonstrate accuracy but require human oversight.

    Jun 19, 2026DOI: 10.1097/hc9.0000000000000983Journal ArticleHepatology Communications

    Written by: Holly K T HuangDebra W YenMichelle Y LiGabrielle JutrasLisa X DengCharles E McCullochMark J PletcherJennifer C LaiBilal HameedJin Ge

    Find these authors on Roon

    JG
    Dr. Jin Ge
    Liver Transplant · UCSF Medical Center

    INTRODUCTION

    Electronic consultations (e-consults) are asynchronous generalist–specialist communication via the electronic health record (EHR) that may eliminate the need for face-to-face specialist visits for low-acuity and low-complexity clinical questions. Prior studies have consistently demonstrated that e-consults significantly improve access to specialist care, reduce wait times, increase cost savings, and enhance provider communication.1–3 These outcomes have been demonstrated in hepatology, with one program achieving a 22-hour average response time compared with 68 days for traditional in-person visits and a reduction of patient travel burden by over 10,000 cumulative miles over 38 months; moreover, primary care providers were able to successfully implement specialist recommendations in the e-consults in 97% of cases.4Despite these promising outcomes in enhancing access to specialist care, the results were mixed for the specialist experience. In prior studies of e-consults, many specialist respondents expressed frustration due to inadequate compensation or time protection for these tasks, liability concerns, and weariness from addressing repetitive low-complexity clinical inquiries.2,5,6 As e-consults become increasingly common in health systems, the “e-consult burden” on specialists like hepatologists is expected to intensify, threatening the viability and sustainability of these programs.7One largely unexplored solution is to use generative artificial intelligence (GenAI) and large language models (LLMs) to expedite the e-consult process through automated drafting of e-consult notes, thereby decreasing burdens on specialists. GenAI and LLM tools have been applied in multiple areas of clinical medicine to enhance diagnostics, medical writing, and clinical decision support.8,9 Moreover, processes like retrieval-augmented generation (RAG) could allow for the construction of specialty-focused LLMs enriched with standardized literature or society-endorsed clinical practice guidance that could be particularly adept at addressing e-consults that require low complexity clinical knowledge.Despite the multitude of use cases, there have been no prior studies specifically validating or comparing the use of LLMs drafting e-consult responses versus human responses. Our group had previously developed LiVersa, a customized RAG-LLM designed specifically for liver diseases. LiVersa is customized utilizing the publicly available clinical practice guidance documents from the American Association for the Study of Liver Diseases (AASLD), which are deployed via RAG on top of protected health information (PHI)-compliant implementations using Microsoft Azure of OpenAI’s family of LLMs at the University of California San Francisco (UCSF).10
    In this study, we demonstrate a proof-of-concept in the use of customized RAG-LLMs for answering e-consultations by evaluating the clinical equivalence, accuracy, and safety of LiVersa’s drafts against a retrospective reference of human hepatologist responses.

    METHODS

    This study was approved by the UCSF Institutional Review Board under Study #25-45586, and all research was conducted in accordance with both the Declarations of Helsinki and Istanbul. We included all e-consults submitted to the adult (age ≥18 y) UCSF Liver Diseases and Liver Transplantation clinic and successfully completed by transplant hepatologists in routine clinical care between January 1, 2025, and March 31, 2025. We excluded all cases that resulted in a recommendation of a video or in-person visit, as these e-consults were not answered. We did not apply other exclusion criteria to maximize generalizability and capture the full diversity of e-consults encountered in clinical practice. The completed e-consult note by human transplant hepatologists is the “gold-standard.”
    We used LiVersa as the baseline RAG-LLM model to generate draft responses to each e-consult. The underlying base LLM used for RAG was the Microsoft Azure OpenAI OpenAI-o1 model. LiVersa received structured clinical data from our EPIC-based EHR system through templated dot phrases. Structured data included complete blood count with differential, comprehensive metabolic panel, liver function tests (aminotransferases, alkaline phosphatase, total bilirubin, gamma-glutamyl transferase), albumin, coagulation studies (INR, PT), Model for End-stage Liver Disease (MELD) score, viral hepatitis serologies and viral loads (hepatitis A, B, and C), iron studies (ferritin, iron, transferrin, total iron-binding capacity, transferrin saturation), autoimmune markers (antimitochondrial antibody, antinuclear antibody with pattern and titer, anti-smooth muscle antibody, immunoglobulins), alpha-1 antitrypsin level and phenotype, ceruloplasmin, recent abdominal imaging (ultrasound, CT within 365 d), echocardiography results, endoscopy reports (esophagogastroduodenoscopy, colonoscopy, endoscopic retrograde cholangiopancreatography, endoscopic ultrasound), radiology results within 90 days, and current outpatient medications. We noted missing data elements as “no results found for”: the specified test. The model did not have access to hepatologist-authored responses during generation. We excluded any clinical information added to the record after the original hepatologist response from the LiVersa input to maintain temporal consistency and avoid information advantage bias.
    We gave LiVersa a standard prompt to generate the e-consult draft (Supplemental Methods S1, http://links.lww.com/HC9/C373). In short, this prompt specified that responses should: (1) be directed to healthcare professionals, (2) consider available diagnostic studies noted in templated e-consult formats, (3) include the most likely diagnosis, (4) remain concise (fewer than 10 sentences), (5) provide actionable next steps including specific diagnostic recommendations, and (6) prioritize virtual management over in-person visits when feasible. When LiVersa deemed an office visit necessary, we instructed it to specify this determination and recommend additional information to collect before the visit.
    We evaluated the content of the LiVersa-generated drafts versus the gold-standard human responses via a semi-structured 12-question survey (Supplemental Methods S2, http://links.lww.com/HC9/C373). We devised this survey in conjunction with and mandated by Artificial Intelligence Governance at UCSF as a standardized measure to evaluate clinical LLM performance, capturing both quantitative and qualitative dimensions. Ten questions used Likert-type scales ranging from 2 to 4 points, with lower scores indicating greater comparability to the reference standard. Eight of the 10 questions (Q1, Q3–Q5, Q7, Q9–Q10, Q12) demonstrated ordinality while 2 of the 10 questions (Q2, Q8) provided categorical information. As the Likert scales for questions varied from 2 to 4 points, we analyzed the 8 questions that demonstrated ordinality after standardizing the scales from 0 to 1, where lower scores (closer to 0) indicate closer resemblance to human responses. The remaining 2 questions (Q6, Q11) elicited free-text responses to provide qualitative feedback on two of the Likert questions, highlighting nuances not captured by Likert scoring, and identifying areas for model improvement. We analyzed these free-text responses descriptively for overarching themes.
    Three transplant hepatologists, who were not involved in authoring the original e-consult responses, independently reviewed and compared LiVersa’s draft responses with the human-authored responses using the survey. We assessed inter-rater reliability using intraclass correlation coefficients (ICCs) based on a 2-way random-effects model for absolute agreement [ICC (2, 3)] with 95% confidence intervals.
    As repeated human assessments of draft quality may not be possible due to time and labor constraints, we also explored the potential of utilizing LLMs for grading through the “LLM-as-a-judge” framework by comparing this process with human assessments.11 We used the OpenAI-o1 model for this purpose without any additional modifications or implementation of an RAG system. We presented OpenAI-o1 with the original consultation question and the hepatologist's reference response. The LLM scored each response using the same survey instrument via a standardized prompt (Supplemental Method S3, http://links.lww.com/HC9/C373). We generated 10 iterations of the LLM-based machine assessment for each case and question. The LLM scored each case and survey question in runs independent of each other to avoid cross-contamination, use of prior context, and snowballing effects. We assessed inter-rater reliability using ICC based on a 2-way random-effects model for absolute agreement [ICC (2, 10)] with 95% confidence intervals.
    We used descriptive statistics to summarize the characteristics of e-consult questions and responses. We used paired t tests to compare the word count, character count, sentence count, and words per sentence between the responses from hepatologists and those from LiVersa. For the survey results, we presented as mean scores with 95% confidence intervals which were calculated using the F-distribution method.12 To compare the scores of human and machine reviewers, we used the Two One-Sided Tests (TOST) procedure with α=0.05. We established an equivalence margin of ±0.1 on the standardized scale; given that individual questions used 2–4 point Likert scales, this margin represents 0.2–0.5 points on the original scales, which we judged a priori as the threshold below which differences would not meaningfully affect interpretation. We excluded questions with descriptive or nominal data from the analysis. We performed statistical analyses using Python (version 3.13.9) with scipy.stats library.

    RESULTS

    A total of 61 e-consults were completed from January 1, 2025, to March 31, 2025, by 8 different hepatologists. Among the cases, 70.5% were female with a median age of 54 years (range, 23–97 y) (Table 1). The most common e-consult question categories were abnormal liver function tests (34.4%), hepatitis B (23.0%), and abnormal imaging (21.3%). The mean clinical question word count was 77 words (SD, 65 words), with variation by category. Questions regarding abnormal imaging and abnormal liver function tests were the most detailed (mean, 109 and 97 words, respectively), while FibroScan or elastography-related questions were the briefest (mean, 29 words). Questions ranged from straightforward test requests (eg, “request for fibroscan for history of hepatic steatosis. Elevated Fib4 score to 1.4”) to complex diagnostic inquiries (eg, “…Please evaluate the FDG-avid lesion and provide recommendations to whether it is suspicious for a process other than sarcoidosis and assess whether a transjugular biopsy is necessary…”) to nuanced management questions (eg, “…If DILI is primary suspect, would it be acceptable from a liver perspective to maintain pt on statin despite low level transaminitis, given benefits of the medication for his severe CAD?…”).
    Characteristics of e-consult requests (n=61)
    Abbreviation: e-consult, electronic consultation.
    We compared metadata of the original gold-standard hepatologist responses and LiVersa-generated drafts (Table 2). Overall, there were no significant differences in word count (284 vs. 264 words, p=0.47), character count (1765 vs. 1798 characters, p=0.86), sentence count (14 vs. 11 sentences, p=0.10), or words per sentence (24 vs. 25, p=0.44). When stratified by consultation category, there were no category-specific differences in word count or words per sentence, with the exception of the category of hepatitis C consults (n=4), where LiVersa-related drafts contained significantly more words (283 vs. 160 words, p=0.01) and words per sentence (27 vs. 17 words per sentence, p=0.01) compared with the gold-standard hepatologist responses.
    Characteristics of e-consult responses by hepatologists versus LiVersa (n=61 pairs)
    Note: p-values are from paired t tests (by case) comparing hepatologist versus LiVersa responses to each e-consult question.
    Abbreviation: e-consult, electronic consultation.
    Three independent hepatologists evaluated the LiVersa-generated drafts using the human responses as reference standards. Between the 3 human raters, they had moderate reliability overall (Supplemental Table S1, http://links.lww.com/HC9/C373). As 8 of the 10 Likert-scale questions that demonstrated ordinality had varied from 2 to 4 points, we analyzed these questions after standardizing the scales from 0 to 1, where lower scores indicate closer resemblance to human responses (Table 3). In comparison to gold-standard human responses, LiVersa-generated drafts had strong, non-inferior safety and accuracy ratings, with better scores for avoiding misleading information (0.07), potential harm (0.12), and extraneous suggestions (0.24). In contrast, LiVersa-generated drafts had weaker scores for utility and comprehensiveness, including clinical equivalence to human responses (0.37), capturing all human suggestions (0.41), producing immediately usable drafts (0.46), and being superior to human responses (0.97).
    Evaluation of LiVersa drafts by 3 human hepatologists
    Notes: Ratings were standardized to a 0–1 scale, where 0 indicates optimal LiVersa performance, and 1 indicates poor LiVersa performance. N=178 ratings per question. Questions with descriptive (Q6, Q11) or nominal data (Q2, Q8) were excluded from analysis.
    Abbreviations: e-consult, electronic consultation; LLM, large language model.
    When assessing for completeness of capture of human suggestions, human reviewers rated 65% of drafts as missing some information (Figure 1). Among their comments, 47% of them described missing studies and/or their interpretation; an example from one reviewer for one LiVersa draft is that it “did not discuss role of IVIG on HBcAb positivity and need to repeat testing” (Supplemental Table S2, http://links.lww.com/HC9/C373). The next common comment is not addressing the consult question (30%); for example, a reviewer noted for one response that “LLM provided general recommendations, such as review the medication list, but did not discuss the likelihood of patient’s medications as cause of elevated liver tests” (Supplemental Table S2, http://links.lww.com/HC9/C373). Twenty percent of comments noted that LiVersa was missing a component of management; for example, a reviewer described that LiVersa “missed resmetirom suggestion, but suggested that if advanced fibrosis should be referred to Hepatology and therefore would have talked about that medication in clinic anyway” (Supplemental Table S2, http://links.lww.com/HC9/C373). Conversely, human reviewers rated most drafts (96%) as not containing any important information that the actual human response omitted (Figure 1). Examples of the few (4%) descriptions of important information that the human response omitted that were included by the LLM include “LLM included information about heterozygosity for hemochromatosis that human response did not mention” and “LLM included frequency of monitoring of HBV/liver tests while on rituximab.”
    Content quality of LiVersa responses as evaluated by human hepatologists and LLM. The figure presents 6 content-quality evaluation dimensions: Q3 (clinical equivalence), Q5 (comprehensiveness), Q7 (precision), Q8 (specificity), Q9 (accuracy), and Q10 (superiority). In each panel, grouped bar charts show the percentage of responses in each response category by reviewer type: human (dark green; n=178 evaluations by 3 human reviewers) and o1 (light green; n=604 evaluations by 10 LLM machine reviewers). Missing scores were excluded from the analysis. In each panel, the x-axis shows the response category for that dimension, and the y-axis shows the percentage of responses (0%–100%). Percentages are displayed on the bars. Abbreviation: LLM, large language model.
    To explore the comparability of assessment between humans and LLMs, we used an LLM (via “LLM-as-a-judge” by a non-customized OpenAI-o1 model) to evaluate the same 61 LiVersa-generated e-consult drafts using the same questionnaire. Overall, among the 10 iterations of the LLM-based machine assessments, the machine reviewers had moderate reliability (Supplemental Table S3, http://links.lww.com/HC9/C373). Human reviewers rated 33% of LiVersa-generated drafts as great starting points, compared with 5% rated by LLM reviewers (Figure 1). In addition, human reviewers rated 20% of LiVersa-generated drafts posing harm to patients without human review, compared with 67% of LiVersa-generated drafts rated by LLM-based machine reviewers (Figure 2). When evaluating content quality, human reviewers rated 48% of the drafts as clinically equivalent, compared with 27% by LLM-based machine reviewers. Human reviewers also rated 83% of drafts as providing appropriate case-specific recommendations, compared with 66% of LLM-based machine reviewers. For 80% of the Likert-scale items, human reviewers rated a greater proportion of LiVersa-generated drafts with the best rating than machine reviewers.
    Utility of LiVersa responses as evaluated by human hepatologists and LLM. The figure contains 4 utility dimensions: Q1 (usability), Q2 (sufficiency of information provided to LiVersa), Q4 (recommendation of clinical visit), and Q12 (implications for harm). In each panel, grouped bar charts show the percentage of responses in each response category by reviewer type: human (dark green; n=178 evaluations by 3 human reviewers) and o1 (light green; n=604 evaluations by 10 LLM machine reviewers). Missing scores were excluded from the analysis. In each panel, the x-axis shows the response category for that dimension, and the y-axis shows the percentage of responses (0%–100%). Percentages are displayed on the bars. Abbreviation: LLM, large language model.
    Equivalence testing between human and machine reviewers demonstrated agreement for straightforward content assessment (accuracy, precision, comprehensiveness; TOST p≤0.05; mean difference range, 0.026–0.029) (Table 4). For more complex judgment assessments (clinical equivalence, recommendation for clinical visit, harm implications, and immediate usability as drafts), LLM-based machine reviewers rated a greater number of LiVersa-generated drafted responses as inferior (as indicated by negative mean difference values) to those of hepatologists (mean difference range, −0.050 to −0.358) (Table 4).
    Equivalence testing of human and LLM-as-judge in evaluating the performance of the LiVersa drafts
    Notes: Ratings were standardized to a 0–1 scale, where 0 indicates optimal LiVersa performance, and 1 indicates poor LiVersa performance. Mean differences represent Human minus LLM-as-judge responses. Negative mean difference indicates worse ratings by LLM-as-judge. p-values are from the Two One-Sided Tests (TOST) procedure with α=0.05, equivalence margin = ±0.1. Equivalence is declared when 90% CI falls entirely within ±0.1. Questions with descriptive (Q6, Q11) or nominal data (Q2, Q8) were excluded from analysis.
    Abbreviations: e-consult, electronic consultation; LLM, large language model.

    DISCUSSION

    In this novel study evaluating the role of LLM in e-consults, we demonstrated the potential for customized hepatology LLMs, such as LiVersa, to serve as drafting tools for e-consults. We included all e-consults completed during the specified time, which comprised a wide variety of e-consult topics and degrees of question complexity to maximize external validity and generalizability. We did not observe substantial differences in length and verbosity between the hepatologist responses and LiVersa responses. This finding supports the feasibility of using LiVersa drafts in some capacity without introducing marked deviations in response style or cognitive load for reviewing clinicians.
    Our results suggest that LiVersa performs more reliably in avoiding factual errors, significant omissions, and misleading recommendations than in achieving excellence or superiority relative to hepatologist responses. This pattern is consistent with LLM’s strengths in synthesizing established knowledge while lacking the contextual judgment and prioritization that characterize expert clinician reasoning. The gap in clinical equivalence further underscores that, while LiVersa drafts may be broadly accurate and safe, they often require human refinement to fully align with clinician intent, patient-specific considerations, and practical decision-making.
    Taken together, these findings support implementation of e-consult-drafting LLMs within a “human-in-the-loop” oversight framework to maximize clinical utility. An approach includes a draft-only deployment model with mandatory human sign-off, allowing clinicians to leverage efficiency gains while maintaining accountability and clinical judgment. Such a model also enables clinicians to selectively edit, contextualize, or override LLM outputs, thereby mitigating risks associated with over-reliance on automated recommendations.
    Our results also showed that “LLM-as-a-judge” assessments were often more conservative than human assessments. LLM-based evaluation performed comparably to human evaluation for metrics grounded in objectivity, including accuracy, precision, and comprehensiveness. In contrast, performance was less concordant for metrics that rely more on clinical judgment, such as clinical equivalence, draft usability, and harm implication. This divergence likely reflects the inherent difficulty of encoding experiential reasoning, contextual prioritization, and risk tolerance by LLMs. Given the conservative nature of LLM evaluation, it has a potential role for rapidly evaluating draft quality. This model can be implemented to automatically grade LiVersa drafts and request human review when the LLM assessment falls below a threshold. It also has the potential utility for evaluating draft quality after a model update (eg, new practice guidelines added to the RAG knowledge set, or a new version of the foundation model) before clinical deployment.
    Our study has the following limitations. First, this study had a limited sample size. Given the labor-intensive process of manual reviews of each e-consult by human hepatologists, only 3 months’ worth of e-consults were included for the study. A larger-scale study following this pilot study can be conducted to confirm our findings. Second, by definition, all included e-consults were deemed appropriate for electronic management by the completing hepatologist; as such, the most complex cases were inherently excluded. As a future direction, it would be important to prospectively evaluate the ability of LiVersa to identify cases unsuitable for e-consultation and appropriately route them to in-person or telemedicine care. The safety and reliability of such triage decisions will be critical for workflow integration. This is planned in our implementation roadmap for the LiVersa e-consult program at UCSF. Third, we acknowledge that the assessment instrument was not formally validated, as this was internally developed with the guidance of UCSF’s Health AI oversight to be consistent with other ongoing AI pilot programs. The use of this assessment limits the strength of the conclusions that can be drawn. Fourth, the inter-rater reliability was moderate for both humans and LLM, with variability across items. This may suggest the inherent subjectivity of assessments, the limitations of the assessment instrument, or both. Considered together, these constraints preclude definitive statements regarding equivalence or quality assurance. For this reason, this study should be viewed as an initial validation, and future work should focus on developing and validating standardized instruments that could be utilized in both internal and external settings with improved reliability. Lastly, given this study’s validation-focused design, the assessments by humans and LLM-as-a-judge were not blinded, which may introduce potential for bias. A fully blinded, head-to-head comparative design incorporating both human and LLM reviewers using a source-agnostic rubric would be an important next step once sufficient baseline validation of LLM performance has been established.
    In summary, this proof-of-concept study demonstrates that a customized, hepatology-specific LLM can generate e-consult drafts that are broadly accurate and stylistically comparable to expert clinician responses while reliably avoiding factual errors and harmful recommendations. Although LiVersa drafts frequently required human refinement to achieve full clinical equivalence, these findings support a human-in-the-loop implementation model in which LLMs serve as drafting assistants under mandatory clinician oversight. LLM-based evaluation also showed promise as a scalable quality-assurance mechanism, particularly for objective metrics, with potential utility in continuous model monitoring and pre-deployment validation. Larger prospective studies using validated assessment instruments will be essential to confirm these findings and establish the evidence base needed for broader clinical adoption.
    JGDr. Jin Ge

    Dr. Jin Ge

    Gastroenterology · Hepatology & Liver Transplantation
    University of California San Francisco Medical Center
    jin