Sample EBS Report
Anonymized data from a representative Oracle EBS 12.2 deployment — the health check, both analyzers and AutoPilot, as TuneVault shows them on your own data.
Oracle EBS 12.2Linux x86_64Paired with a 19c database tierSample data — representative run
72
/100
Health Score
2
Critical Findings
2
Warnings
11/15
Checks Passed
Online
App Tier Status
TuneVault AI SummaryGPT-4o powered
Two problems your users will feel today. Workflow email has stopped: the Notification Mailer stopped itself after repeated errors and 1,044 notifications are waiting. The Workflow Mailer Analyzer walked the whole mail path and found everything upstream healthy, so a restart on its own would fail the same way — fix its outbound server first. XX AR Aging Extract is running 9× slower than its own history: the Request Analyzer traced it to a SQL plan that changed five days ago, and the faster plan can still be restored. Separately, the APPS schema has 734 invalid objects. AutoPilot restarted Apache/OHS once overnight; every other component is healthy — OPMN, WebLogic AdminServer, Node Manager, Forms, OACore and the Concurrent Managers (21/21 processes at target).
⚡
Top Action Now
Read the mailer's container log for the exception, correct its outbound server, then Reset it (EBS Ops → WF Mailer → Reset). Then restore the aging extract's previous plan with a SQL Plan Baseline for SQL 7fz2m0kq9w3xa (plan 1298807715).
The mailer stopped itself after its errors exceeded its error budget, so a restart on its own fails the same way. The Workflow Mailer Analyzer (Workflow Mailer tab) found every layer upstream healthy and all 1,044 emails parked on WF_NOTIFICATION_OUT: the fault is the mailer itself, most likely its outbound server.
Resolution steps:
1. Read the exception in the mailer's container log — the newest $APPLCSF/$APPLLOG/FNDCPGSC<pid>.txt for the Workflow Mailer Service
2. Correct the outbound server, and the context variable s_smtphost so AutoConfig keeps it
3. Reset the mailer: EBS Ops → WF Mailer → Reset (stops it, clears the status, starts it)
4. Expect 212 Framework notifications to render through oacore as the backlog drains
-- What Reset does (as APPS) — only after the cause is fixed:
UPDATE fnd_svc_components
SET component_status = 'STOPPED', component_status_info = NULL
WHERE component_type = 'WF_MAILER';
COMMIT;
-- Then use OAM to Start, or via PL/SQL:
DECLARE
l_retcode VARCHAR2(100);
l_errbuf VARCHAR2(4000);
l_cid NUMBER;
BEGIN
SELECT component_id INTO l_cid FROM fnd_svc_components
WHERE component_type = 'WF_MAILER' AND ROWNUM = 1;
fnd_svc_component.start_component(l_cid, l_retcode, l_errbuf);
DBMS_OUTPUT.PUT_LINE('retcode: ' || l_retcode);
END;
/
🔴 CriticalAPPS Schema — 734 Invalid Objects
734 objects▾
734 objects in the APPS schema are INVALID. This typically indicates a failed or incomplete patch application, or a DDL change to a base object that invalidated dependent objects. Invalid objects cause ORA-04063 errors at runtime when the affected code path is executed.
-- Check count by type:
SELECT object_type, COUNT(*) cnt
FROM dba_objects
WHERE owner = 'APPS' AND status = 'INVALID'
GROUP BY object_type ORDER BY cnt DESC;
-- Recompile via AD utility (as applmgr):
-- sqlplus apps/<pwd> @$AD_TOP/sql/adcompsc.sql APPS
-- Or via DBMS_UTILITY:
EXEC DBMS_UTILITY.COMPILE_SCHEMA('APPS', FALSE);
⚠ WarningOptional Managed Servers — Not Running
OAFM not started▾
oafm_server1 is not running. It hosts EBS web services such as Integrated SOA Gateway; if you do not use them, this is expected. Start it with admanagedsrvctl.sh start oafm_server1 when they are needed.
⚠ WarningDB Tablespace — APPS_TS_TX_DATA at 81.7%
autoextend OFF▾
Reported from the paired DB-tier connection. APPS_TS_TX_DATA is the primary EBS transaction tablespace at 81.7% with autoextend disabled. Add a datafile or enable autoextend on the DB server before the next batch window.
EBS App Tier Check Results
Check
Status
Detail
Apache / OHS
RUNNING
adapcctl.sh status — OK, PID 12481
OPMN
RUNNING
adopmnctl.sh status — alive (PID 12523)
Apps Listener (adalnctl)
RUNNING
lsnrctl TCP status — LISTENER: ready
Node Manager
RUNNING
adnodemgrctl.sh status — running (PID 13012)
WLS AdminServer
RUNNING
WLST domain runtime — AdminServer: RUNNING
OACore Managed Servers
RUNNING
oacore_server1 RUNNING, oacore_server2 RUNNING
Forms Managed Servers
RUNNING
forms_server1 RUNNING, forms_server2 RUNNING
Concurrent Manager (adcmctl)
RUNNING
21/21 worker processes at target, 1 pending request
Workflow Notification Mailer
DEACTIVATED_SYSTEM
147 errors, 1,044 pending — restart required
EBS DB Health (from paired DB connection)
Check
Status
Detail
APPS Invalid Objects
CRITICAL
734 invalid APPS objects — recompile required
APPS_TS_TX_DATA Tablespace
WARNING
81.7% used, autoextend disabled
System Resources
✅
Disk Usage
68% — OK
✅
Memory
72% used — OK
✅
CPU
Avg 4.1% — OK
✅
I/O Wait
3.2% — OK
Request Analyzer answers “why is this concurrent request taking so long?” It lists running requests that are slower than their own program's history, then walks every layer on both tiers — the request, its database session, where the time goes, the SQL and its plan history, the database as a whole, the concurrent node and the request's process on it, the network between them, and external dependencies — before giving an evidence-backed verdict with a confidence level. Read-only. Every paid plan.
Long-Running Request Analyzer
Request 4823917 · XX AR Aging Extract · running 2 h 48 m on ebs-cp01 · both tiers walked: database + app-node agent
9 layers · 38 steps29 checked3 findings8 not used herewalk took 44 s
Now running plan 3561904772 at 2 h 48 m and counting for its one execution
Plan 1298807715 ran it at ~14 min/exec (last seen 6 days ago) — 12× faster so far
Every object plan 1298807715 used still exists (4.6)
What to do
Pin the previous plan with a SQL Plan Baseline loaded from AWR (plan 1298807715), then find what changed — statistics or bind values. Test before applying in production.
AI summary
The aging extract is not stuck: it is working hard on the wrong plan. Its main query switched plans five days ago, the day after statistics were last gathered on AR_PAYMENT_SCHEDULES_ALL, and the new plan walks the payment schedules row by row from every customer account. The previous plan finished in about 14 minutes and every object it used still exists, so a SQL Plan Baseline can restore it. Nothing else is in the way: no blocker, a quiet database, a healthy app node and network.
The walk — every step, every run
✓ healthy⚠ finding– not used by this request○ not needed⊘ not yet available
RequestWhat it is, how long it usually takes, and whether it is waiting on something of its own
✓1.1 Request contextXX AR Aging Extract (XXAR_AGING_EXT) · request 4823917 · submitted by AR_BATCH · running 2 h 48 m
Which program this is, who ran it, its parameters and how long it has been running. Everything after builds on this.
⚠1.2 Runtime vs this program’s historySlower than usual: its last 30 completed runs took 18 m median, 26 m at the 90th percentile — this run is already 9.3× the median
Compares this run with the program's own completed runs. A three-hour job is normal for some programs and alarming for others; 'slow' is only meaningful against its own history.
✓1.3 Program typePL/SQL stored procedure — the work happens in the database
SQL*Plus, PL/SQL, Reports, Java or a shell script. It decides where the time can go: a host or Perl program does its work on the app node, not in the database.
–1.4 Child requestsNo child requests
A parent request often just waits for its children. If one child is still running, that child is what to analyze, not the parent.
✓1.5 Request statusRunning / Normal
Paused, Terminating or Resuming each mean something different from Running, and change what the other checks can show.
✓1.6 Concurrent managerStandard Manager process on ebs-cp01 is alive and owns the request
The concurrent manager process that owns the request. If that manager is gone, the request can be left behind as an orphan that nothing will ever finish.
✓1.7 Still running at the end?Still running when the walk finished
Re-read at the end: if the request finished during the analysis, the findings describe it while it was still running.
Database sessionFind the session doing the work and read what it is doing right now
✓2.1 Find the database sessionSession 1873,40121 on instance 1, matched from the process id FND recorded
Finds the database session doing the work, from the process and session ids the concurrent manager recorded. No session at all usually means the work is outside the database, or the connection was lost.
What the session is doing right now: working, waiting and on what, and for how long. The first clue to where the time goes.
–2.3 Progress reported by OracleNo long operation reported — index lookups inside a nested loop never appear here
Oracle's own progress report for long operations, such as full scans and sorts, with an estimate of the time remaining when it has one.
Where the time goesTime profile of the session over the recent past
✓3.1 Last 30 minutes (ASH)Last 30 min: 94% in SQL 7fz2m0kq9w3xa · 77% db file sequential read, 21% CPU (Diagnostics Pack confirmed)
The last 30 minutes of where this session spent its time, from Oracle's activity history. Needs the Diagnostics Pack, so it is used only when the pack is confirmed.
○3.2 Live samplingASH was available, so live sampling was not needed
Without the Diagnostics Pack, the session is sampled live several times instead. Less history, same question: where is the time going now?
✓3.3 Is it doing work?Working: 1.84 M buffer gets and 212 K physical reads between two reads 20 s apart
Reads the session's work counters twice. Rising counters mean it is working; frozen counters mean it is only waiting. A request can look Running and be doing nothing.
SQLThe statement taking the time, its plan, and whether that plan changed
⚠4.1 SQL taking the timeSQL 7fz2m0kq9w3xa (SELECT … FROM AR_PAYMENT_SCHEDULES_ALL ps, HZ_CUST_ACCOUNTS ca …) — 1 execution, 41.2 M buffer gets so far
The statement taking the time, with how often it ran and how much it read. Inside a PL/SQL program the statement that matters is often not the first one.
✓4.2 Execution planPlan 3561904772: NESTED LOOPS driving from HZ_CUST_ACCOUNTS into AR_PAYMENT_SCHEDULES_N2
How Oracle is executing that statement right now: which tables, which indexes, and in what order.
⚠4.3 Plan historyPlan 1298807715 ran this statement at ~14 min per execution across 30 runs, last seen 6 days ago; plan 3561904772 first appeared 5 days ago
If this statement ran faster before with a different plan, the slowdown is a plan change, not more data. Needs the Diagnostics Pack.
✓4.4 Optimizer statisticsStatistics present and recent: AR_PAYMENT_SCHEDULES_ALL and HZ_CUST_ACCOUNTS last analyzed 5 days ago
Statistics tell the optimizer how big the tables are. Stale or missing statistics are the commonest cause of a sudden bad plan.
✓4.5 Index coverageEvery filter column in the plan leads an existing index
Compares the columns the plan filters on with the indexes that exist. A full scan with no index on its filter column is the missing-index case.
✓4.6 Does the faster plan still exist?Every object plan 1298807715 used still exists
If the faster past plan used an index, checks that index still exists. A dropped index explains a plan regression better than anything else.
Database healthIs another session in the way, and is the database as a whole healthy right now?
✓5.1 Blocking sessionNo blocking session
Is another session holding a lock this one needs? Names the blocker, and the user or concurrent request behind it.
✓5.2 Contention in this sessionNo contention waits in this session’s time profile
Contention waits (latches, buffer busy, commits) inside this session, which point at other work competing for the same resources.
✓5.3 Database load right nowDatabase load without this request: 2.9 active sessions on 16 CPUs
How busy the whole database is right now, minus this request's own load. A slow request on an overloaded database is a victim, not the cause.
✓5.4 Database hostDatabase host: 21% CPU busy, no paging
CPU, load, memory and paging on the database server itself. A host that is swapping slows everything on it.
Remote databaseIs the time spent waiting on a database link?
–6.1 Database link waitsNo database link waits
Time spent waiting on a database link is time spent in another database. That remote database, or the network to it, is where to look.
App nodeThe concurrent-processing node and the request’s OS process on it
✓7.1 Agent on the request’s nodeTuneVault agent found on ebs-cp01.example.com, the node running the request
Finds the TuneVault agent on the node the request runs on. Both tiers are required, because the cause can be on either side.
✓7.2 Node healthLoad 1.4 on 8 CPUs · memory 58% · swap 0 · log/output filesystem 61% used
CPU, load, memory, swap and free space on the concurrent-processing node. An overloaded node slows every request on it.
✓7.3 OS process alive?The request’s process is alive
Is the request's own OS process still alive? A request whose process is gone is an orphan that only an administrator can clean up.
✓7.4 Process CPU trend0.3 s of CPU in 20 s — idle on the node, as expected while the database works
CPU used by the request's process, read twice. Busy on the node but idle in the database means the work is in the program itself.
✓7.5 Log / output growthLog unchanged — this program writes its output when it finishes; the database counters (3.3) show it working
Whether the log and output files are still growing. A request writing nothing, using no CPU and doing no database work is hung.
NetworkThe connection between the app node and the database
✓8.1 DB connection of the processOne ESTABLISHED connection from the process to the database listener
The process's own network connection to the database. If the process is alive but has no connection, the database session was lost.
✓8.2 App node → database round tripTCP connect to the listener from the node: 0.6 ms average over 5 tries
Timed connections from the node to the database listener. A slow or failing network makes every call slow, however healthy each side is.
✓8.3 Session vs process agreementDatabase session and OS process agree
The database session and the OS process must agree. Process alive with no session, or session alive with no process, explains a request that never finishes.
External dependencies & outputLinks, files, vendor calls, BI Publisher, bursting and delivery — each checked when the request uses it
–9.1 Database linksNo database links in the plan or the SQL text
Database links found in the plan or the SQL text, and which remote database each one reaches.
–9.2 External tables & filesNo external tables in the plan
External tables read files from the database server's file system. A missing file, mount or directory stops the request.
–9.3 Outbound calls (vendors, web services)No socket waits and no non-database connections
Calls out to other systems, such as vendors and web services, show as socket waits or connections that are not to the database. The other system is then where the time goes.
–9.4 Output post-processing (BI Publisher / OPP)The program has no BI Publisher template
After the program finishes, the Output Post Processor builds the BI Publisher output. If OPP is down or busy, the request waits there.
–9.5 Bursting & deliveryNo bursting or delivery actions
Bursting splits the output and delivers it by email, print or file. Each delivery is another step the request waits for.
✓9.6 Child processes on the nodeNo child processes under the request’s process
Processes started by the request, such as sqlplus, shell and Java children. The child doing the work is sometimes the one that is stuck.
Workflow Mailer Analyzer answers “where does EBS Workflow email stop?” It walks the whole mail path every time — Generic Service Management, the containers, the mailer and its listeners, the event system, the queues, object integrity, recipients, the mailer's configuration, housekeeping and the oacore that renders notifications — and can trace a single notification to the stage where it stopped. It never reads an encrypted mailer value and never sends mail. Every paid plan.
Workflow Mailer Analyzer
Workflow Notification Mailer · the whole mail path, W1–W11 · database + app-node agent (no WebLogic password)
11 layers · 41 steps38 checked8 findings3 not yet availablewalk took 31 s
W1 GSM & containers✓ checked, healthy
W2 Service components⚠ finding
W3 Control channel & events✓ checked, healthy
W4 Queues⚠ finding
W5 Object integrity✓ checked, healthy
W6 Notifications & recipients⚠ finding
W7 Mailer configuration✓ checked, healthy
W8 Node: connectivity? not fully checked
W9 Node: logs? not fully checked
W10 Housekeeping & volume⚠ finding
W11 oacore & the HTML agent⚠ finding
Verdict
The Notification Mailer stopped itself after repeated errors HIGH CONFIDENCE
Evidence
Workflow Notification Mailer: System Deactivated
Error budget: 10 errors per processor before it stops itself
1,044 emails waiting for it on WF_NOTIFICATION_OUT, oldest 2 d 6 h
What to do
The mailer stops itself when its errors exceed that budget, so restarting it alone fails again the same way. Fix the cause first — its container log is the newest $APPLCSF/$APPLLOG/FNDCPGSC<pid>.txt for the Workflow Mailer Service; the node log check (Phase 2) reads it for you (R1). Then Reset it (EBS Ops → WF Mailer → Reset: stops it, clears the status, starts it) (R2). Raising the error count only hides the cause.
Also found
Recent notifications are waiting for email — 1,044 notification(s) from the last 7 days waiting over an hour for email
Notification rendering and your users share one oacore — 212 Framework notifications waiting · the mailer renders through the same oacore your users use
Recipients who can never receive email — 3 user(s) with notification preference DISABLED
Not checked
8.1, 8.2 SMTP / IMAP from the node — Phase 2
9.1 Mailer container log — Phase 2
AI summary
Everything upstream of the mailer is healthy — GSM, the containers, the event system, the listeners and the queues — and every waiting email is parked on WF_NOTIFICATION_OUT, so the fault is the mailer itself, not Workflow. It stopped itself after exceeding its error budget, which means a plain restart will fail the same way. Its outbound server is the concurrent node's own name, a classic cause when the relay on that node accepts only localhost connections: read the mailer log first, fix the server, then Reset. Expect a burst of 212 oacore renders when it resumes, on the same oacore your users are on.
The walk — every step, every run
✓ healthy⚠ finding– not used by this mailer○ not needed⊘ not yet available
GSM & containersGeneric Service Management, the Service Manager, and the two containers the mailer and its listeners run in
✓1.1 GSM enabledGSM Enabled = Y
Every Workflow service runs under Generic Service Management. If this profile is N, no container starts and no mail moves at all, whatever the mailer's own settings say.
✓1.2 Service ManagerService Manager on ebs-cp01: 1 of 1 process running
The Service Manager starts the service containers on each node, and runs with the concurrent managers. When it is down, concurrent processing on that node is usually down too: the mailer is a symptom, not the cause.
✓1.3 Mailer and listener containersWorkflow Mailer Service 1/1, Agent Listener Service 1/1, Document Web Services 1/1 — all on ebs-cp01
The mailer runs in the Workflow Mailer Service; the listeners that feed it run in the Agent Listener Service. A container with 0 processes runs nothing inside it, and every component status below is only the last thing it recorded.
Service componentsThe Notification Mailer and every agent listener: configured, running, or stopped by their own errors
The component that actually sends email. Its status text is its own explanation: 'restarted the maximum number of times' means it stopped itself after repeated errors, so restarting without fixing the cause fails the same way.
Moves each new notification from WF_DEFERRED onto the mailer's queue. If it is stopped, another running listener may still be covering for it; this check says which, so mail that still flows is not reported as lost.
✓2.3 Other agent listenersError, inbound and Java listeners: Running
The error, inbound and Java listeners. Inbound processes email replies; the Error listener routes Workflow errors to the administrator. One stopped on purpose (User Deactivated) is reported as that, not as a failure.
Control channel & eventsThe Business Event System and the subscriptions that put a notification on the mail path
✓3.1 Local system statusLocal system Enabled
The Business Event System dispatches events only while the local system is Enabled. A disabled system silently stops notifications from ever reaching the mail path.
The subscription that puts each notification on the mail path. A clone's post-clone script disables it on purpose, so a copy cannot email real users: the commonest reason a refreshed test system sends no mail.
✓3.3 Control channel subscriptionControl channel subscription Enabled
Containers receive start, stop and refresh commands through this subscription. With it off, starting or stopping the mailer from OAM appears to do nothing.
✓3.4 Control Queue CleanupControl Queue Cleanup ran 4 h ago, scheduled every 12 h
Removes subscribers left behind by containers that no longer exist. Without it the control queue grows and container commands slow down. Oracle recommends every 12 hours.
QueuesWhere messages are waiting — the single most useful fact about which part is broken
✓4.1 Queues enabledAll Workflow queues enabled for enqueue and dequeue
A queue disabled for enqueue or dequeue stops everything behind it, and nothing else reports an error.
⚠4.2 Where messages are waitingWF_NOTIFICATION_OUT: 1,044 READY, oldest 2 d 6 h · WF_DEFERRED: 0 waiting — mail is stuck AT the mailer, not before it
Where a message waits says which part is broken: waiting on WF_DEFERRED points at the listener, on WF_NOTIFICATION_OUT at the mailer or the mail server. The single most useful fact in this analysis.
✓4.3 Expired and undeliverable messagesNo messages expired or moved to the exception queue in the last 7 days
A message that fails every retry moves to the exception queue and is never sent again on its own. Old ones are history; recent ones mean a problem happening now.
✓4.4 Job queue processesjob_queue_processes = 40
Queue propagation and the queue time manager run as database jobs. Below 2 processes, messages can sit READY while every service shows Running.
✓5.1 Invalid Workflow objectsNo invalid Workflow code on the mail path (the invalid APPS objects on the EBS Checks tab are elsewhere)
Invalid Workflow code on the mail path (the notification, mail and event packages) fails every notification that touches it. Invalid objects elsewhere are listed but kept apart, so they raise no false alarm.
✓5.2 IndexesWF_NOTIFICATION_OUT_N1 VALID; no unusable Workflow indexes
WF_NOTIFICATION_OUT_N1 lets the mailer dequeue by correlation. Without it the mailer slows to a crawl as the queue grows.
APPS reaches Workflow objects through synonyms. One that points at nothing breaks every caller, usually after a failed patch or clone.
✓5.4 Queue tables and viewsEvery Workflow queue table and AQ$ view present
Each queue needs its queue table and its AQ$ view. A missing one means a damaged install: an Oracle Support case, not something to recreate by hand.
✓5.5 SequencesEvery Workflow sequence present
Notification and error ids come from these sequences. A missing one stops new notifications from being created at all.
Notifications & recipientsNotifications waiting for email, and recipients who can never receive it
⚠6.1 Notifications waiting for email1,044 notifications from the last 7 days waiting for email, the oldest for 2 d 6 h
Notifications marked for email that have not gone out. Only the last 7 days count as a current problem; long-lived systems carry years of old ones.
⚠6.2 Failed and errored email3 FAILED in the last 7 days — all addressed to recipients whose email is DISABLED (6.3)
Email the mailer tried and could not send. A recent rise means something changed: the mail server, an address, or the content.
⚠6.3 Recipients with email disabled3 users with notification preference DISABLED
The mailer sets a user's preference to DISABLED after their email bounces. From then on they receive nothing, and nobody is told.
✓6.4 Recipients with no addressEvery recent recipient with a mail preference has an address
A recipient who wants email but has no address can never receive one.
✓6.5 Workflow administrator reachableWorkflow administrator role has an address and a mail preference
Workflow errors are notified to this role. If it cannot receive email, failures go unnoticed.
Mailer configurationOutbound and inbound servers, override address, links, threads — never an encrypted value
✓7.1 Outbound serverOUTBOUND_SERVER = ebs-cp01.example.com (a node name — works only if that node relays mail; checked from the node in W8)
Where the mailer connects to send. AutoConfig rewrites it from s_smtphost, and a node name works only if that node relays mail. A relay listening on localhost only refuses every connection to the node name, and the mailer stops itself after enough errors.
✓7.2 Override addressNo override address set
When set, every email goes to this one mailbox instead of its recipient. Right on a test system; on production it means nobody gets their mail.
✓7.3 Inbound configurationOutbound-only mailer (no inbound server) — valid
Approving or rejecting by email reply needs an IMAP server, separate folders and one inbound thread. An outbound-only mailer is valid.
✓7.4 Links in the emailHTMLAGENT matches APPS_FRAMEWORK_AGENT
Links and images in the email, and every Framework region in its body, come from the HTML agent: oacore behind OHS. A wrong host breaks the links; an unhealthy oacore breaks the rendering.
✓7.5 Threads and error budget1 outbound thread · error budget 10 errors per processor
Outbound threads at 0 means nothing is ever sent. The error budget is how many errors the mailer tolerates before it stops itself.
Node: connectivityFrom the concurrent-processing node: can it reach the mail servers and the HTML agent?
⊘8.1 SMTP server reachable from the nodePhase 2 — not run from the node yet
From the node the mailer runs on: can it reach the mail server, does TLS verify, what does the server accept. The database cannot see any of this.
⊘8.2 IMAP server reachable from the nodePhase 2 — not run from the node yet
From the same node: can it reach the IMAP inbox the mailer reads email replies from. If not, approvals sent by email reply are never processed, while outbound mail keeps working.
Node: logsThe mailer container’s own log — the exact exception behind a status
⊘9.1 Mailer container logPhase 2 — not read yet; the log is the newest $APPLCSF/$APPLLOG/FNDCPGSC<pid>.txt
The mailer's own log names the exact exception behind a status, and it is the only place an email lost after EBS marked it SENT shows up.
Housekeeping & volumeThe programs that keep Workflow moving, and how much it is carrying
✓10.1 Workflow Background ProcessWorkflow Background Process ran 11 times today, every 15 min
Deferred activities, timeouts and stuck processes only move when this program runs. Unscheduled, approvals wait forever.
✓10.2 Purge Obsolete Workflow Runtime DataPurge Obsolete Workflow Runtime Data ran 6 days ago, weekly
Without it closed workflows are never removed, the Workflow tables grow, and everything that reads them slows down.
⚠10.3 Errored workflow activities147 errored workflow activities in the last 7 days
Workflow steps that failed. Recent ones are problems happening now; old ones are history.
✓10.4 Workflow volume48,210 open items · 1.9 M notifications
How much Workflow is carrying, and how much of it could be purged.
oacore & the HTML agentThe mailer builds every Framework notification by fetching it from oacore — and that load lands on the oacore your users share
✓11.1 Application JVMs on the database6 application JVMs hold database sessions: 2 oacore, 2 forms, 1 oafm, 1 Workflow container
Every application JVM (oacore, forms, oafm, the Workflow containers) holds database sessions that record its OS process id. That is how oacore is found without a WebLogic password.
✓11.2 oacore on the nodeoacore_server1: 1.6 GB resident, 9% CPU, up 9 d · oacore_server2: 1.5 GB, 7% CPU, up 9 d (heap limit not read)
oacore renders every Framework region in a notification. Out of memory, a runaway request or too many users make those renders time out, the mailer logs errors, and enough errors stop it. A restart in the last hour can be an OutOfMemoryError.
✓11.3 HTML agent answers from the nodeHTML agent accepts connections from ebs-cp01
The mailer fetches notification content and builds every link from this address. If it does not answer, Framework notifications fail and links in every email are broken.
⚠11.4 Notifications that need oacore212 of the waiting notifications carry a Framework (JSP:) region — 212 oacore renders the moment the mailer starts
Each one is an oacore page render. A burst (a mass approval, a backlog flushing after an outage, a bulk resend) can slow oacore for every self-service user.
⚠11.5 Mail and users on one oacore?Mailer HTMLAGENT = APPS_FRAMEWORK_AGENT: mail and your self-service users share one oacore
If the mailer renders through the same oacore your users use, mail and users slow each other down. Oracle recommends a separate oacore for heavy notification volume.
AutoPilot watches this app tier around the clock: a scheduled health check, a 5-minute service heartbeat on the real server, and — for the services you opt in — a restart after a run of consecutive failed readings. It pauses itself during ADOP patching, never restarts the same service twice inside its cooldown, and logs every action with the command's own output. Auto-restart: Business plan and above.
Health Check Schedule
Enabled
Cadence: Every 1 hour Last run: 12 min ago · score 72 Alerts: new findings only → [email protected] TuneOps tickets: auto-created
Service Heartbeat
7 services up
Polled every 5 min on the server itself Last poll: 2 min ago
Service Auto-Restart
On · 5 services
Trigger after 3 consecutive down readings (≈15 min) Cooldown: 30 min per service Paused while ADOP is patching
Service Heartbeat
Service
Status
Last Checked
Consecutive Down
Auto-Restart
Apache / OHS
🟢 Up
2 min ago
—
Opted in
OPMN
🟢 Up
2 min ago
—
Opted in
Apps Listener
🟢 Up
2 min ago
—
Opted in
Node Manager
🟢 Up
2 min ago
—
Opted in
WLS Admin Server
🟢 Up
2 min ago
—
Monitor only
OACore managed servers
🟢 Up
2 min ago
—
Opted in
Forms managed servers
🟢 Up
2 min ago
—
Monitor only
Managed servers are discovered by their real names on each run, and each one that is down is restarted on its own. Optional groups that are not deployed are never restarted. The Workflow Notification Mailer is not an auto-restart target: the Workflow Mailer tab explains why this one stopped.
AI analysis, EBS component checks, both analyzer walks and AutoPilot — every tab in one document, no watermark. Saves as PDF from your browser's print dialog.
Your EBS report will look exactly like this
Ready to run this on your EBS instance?
Connect your EBS app tier. Health check runs in under 5 minutes — no SSH credentials required.