In order of severity:
1) Add a CHECK() that a pointer is not nullptr, where the analyzer
believes this is possible.
2) Add `final` appropriately to functions called from constructors.
3) Add missing cloexec flags.
4) Add missing `noexcept` and other subtle performance warnings
Test: build with clang-tidy
Change-Id: Ifd9a1299a51027a47382926b2224748b5750d6cf
Other log buffers may not use LogBufferElement, so we should decouple
the two classes. This uses an intermediate LogStatisticsElement
structs instead of passing a large number of parameters to each
function.
This additionally moves IsBinary() and the GetTag() functions out into
LogUtils.h since they can be used generically by other users.
Test: logging unit tests
Change-Id: I71f53257342c067bcccd5aa00bae47f714cd7c66
This is required for tests that are aware of sequence numbers to pass;
each new LogBuffer instance should start from sequence = 1, which
isn't the case if the current sequence number is a static.
Test: unit tests
Change-Id: Ie488f8ac5e22b946b7e6237d1d5caf14929c0ec3
This saves 4 or 8 bytes off of each log message for 32 bit or 64 bit
devices respectively. In practice, this actually saves more, due to
avoiding heap fragmentation.
Averaging over 5 runs of the LogBufferTest.random_messages unit test
(32 bit), this change results in 8k less memory used when 1000 logs
are logged and results in 260k less memory used when 10000 logs are
logged.
Test: check memory usage during LogBufferTest.random_messages
Test: logging unit tests
Change-Id: Ia7953e3c4cb19631ef43bab1deb91bb336bc2520
class LogCommand isn't needed, so remove it. Since the rest of
LogCommand.cpp only has to do with permissions, rename it
appropriately.
Test: logging unit tests
Change-Id: I32d09c74d00b6e50083e46832eca3dd886b46682
In the future, we'll want to be able to write to outputs that are not
necessarily a libsysutils SocketClient, for example host tests of
LogBuffer. Therefore, we add a LogWriter class to be used instead of
SocketClient.
Test: logging unit tests
Change-Id: I4385be65e14e83a635691a7ba79e9bf060e49484
We may use different implementations of LogBuffer in the future, so we
make it interface and create a concrete ChattyLogBuffer class that
implements it.
Test: logging unit tests
Change-Id: I5731d6404640664c9acc26b7c677dff3110c6a11
LogStatistics is intertwined with LogBuffer, even relying on it for
thread safety. This needs to change to have a proper
LogBufferInterface, so this CL separates them. Specifically:
1) Adding a lock to LogStatistics and adding thread annotations to
ensure that data structures are protected appropriately.
2) Moving prune_rows calculation into LogStatistics so it is done
while holding this lock.
3) Using LogStatistics instead of LogBuffer where appropriate.
Note that there should not be a significant performance regression
with this lock, as it will almost always been uncontended. If
anything, it should alleviate pressure from LogBuffer's lock.
Test: logging unit tests
Change-Id: I9d6dde2c96c9f024fa0341711c7bc63379e8e406
log/log.h primarily concerns itself with writing logs. The few users
who read logs should directly include log/log_read.h.
Bug: 78370064
Test: build
Change-Id: Ie95c55ea2ffc76fc95768323d445ada6ad4f2520
This reverts commit 5a34d6ea43.
There is a long standing bug that logd will leak memory during its
prune process if the time on the device changes significantly forwards
then backwards. This is due to using the timestamp of each log
message to determine what log messages are yet to be processed by a
reader thread.
Various attempts have been made to rectify this, but the only solution
that safely fixes this issue is to go back to using sequence numbers
on the log messages.
Bug: 64675203
Bug: 77971811
Bug: 149340579
Bug: 150923384
Test: logcat output looks sane
Change-Id: Ibce79cf184eb29a4914f3e42a8cb2868d04dc165
logger_entry and logger_entry_v2 were used for the kernel logger,
which we have long since deprecated. logger_entry_v3 is the same as
logger_entry_v4 without a uid field, so it is trivially removable,
especially since we're now always providing uids in log messages.
liblog and logd already get updated in sync with each other, so we
have no reason for backwards compatibility with their format.
Test: build, unit tests
Change-Id: I27c90609f28c8d826e5614fdb3fe59bde22b5042
logd currently only reports the UID of a log message for 'privileged'
readers (those with a uid or gid of root, system, or log). However,
UIDs are not particularly sensitive. Much more importantly,
non-privileged readers can only see less messages from their own UID,
so this restriction is essentially a no-op, as those readers will
already know their own uid.
Test: liblog and logd unit tests
Change-Id: I9da7d15eb840ba3200128391e70d618eec79f988
rprichard@ pointed out a bug where LogBufferElement::setDropped()
reallocates mMsg to the size of mMsgLen in the case where getTag() !=
0. However, mMsgLen is in a union with mDroppedCount and
mDroppedCount is the value used when a message is already dropped.
Therefore, it's possible that logd uses the wrong value and allocates
much more memory than intended.
We do call setDropped() on elements that have already been dropped in
LogBufferElementLast::coalesce(), so this is not a superfluous issue.
To simplify this even more, this code puts an mTag in a union with
mMsg, such that if mDropped is true, there will never be an allocated
message; mTag will be directly referred to instead. This also reduces
the number of allocations needed very slightly.
Test: logd/liblog/logcat unit tests.
Change-Id: Ia1bfba076439fe31c745a243283d41902bca45ac
LogBufferElement copy constructor supported only partial function.
Solution: handle all cases.
Test: unit test for calling the copy constructor with all possible states.
Change-Id: I55091569d98eb35a09b4c3fc068836ecd256558c
- Remove virtual from the destructor.
- Remove mTag and derive it when calling getTag().
- Add a boolean mDropped to tell when a message is dropped.
- When dropping a message, and it contains valid tag data, reallocate
the message to only contain the tag data.
- Add the packed tag to the class.
This saves about ~150K of PSS on a typical log stream since it moves the
size of the LogBufferElement from 48 bytes to 32 bytes which puts it in
a smaller bin.
Bug: 63792187
Test: Builds, unit tests pass.
Change-Id: Ia5afce343ea3d344fcecd78c648338e94f5c9312
Switch to a reader writer lock for the Element List lock. Also setup
for a reader writer lock for the Times list, but continue to use a
mutex where rdlock() and wrlock() are the same implementation for now.
This should improve general reader performance and prevent blocking of
other reader operations or exit by a single hung logd.reader.per
thread. For example, a full length logcat of an empty buffer (eg:
crash log buffer) will hold a lock while the iterator scans the entire
list.
Test: gTest liblog-unit-tests, logd-unit-tests, logcat-unit-tests
Bug: 37378309
Bug: 37483775
Change-Id: If5723ff4a978e17d828a75321e8f0ba91d4a09e0
If the last buffer has zero length, strip it out of the iovec
issued to SocketClient::sendDatav().
Test: gTest liblog-unit-tests, logd-unit-tests, logcat-unit-tests
Bug: 36497967
Change-Id: I8fc585bbec63402d0e818ff4c620fdd7edcc38dc
Use getRealTime() instead and leverage private liblog log_time
comparison and math functions. This saves 8 bytes off each
element in the logging database.
Test: gTest liblog-unit-tests logd-unit-tests logcat-unit-tests
Bug: 35373582
Change-Id: Ia55ef8b95cbb2a841ccb1dae9a24f314735b076a
Switch _all_ file's coding style to match to ease all future changes.
SideEffects: None
Test: compile
Bug: 35373582
Change-Id: I470cb17f64fa48f14aafc02f574e296bffe3a3f3
Report multiple identical chatty messages differently than for
regular expire chatty messages. Multiple identical will
report identical count, while spam filter will report
expire count.
This should reduce the expected flood of people confused
but chatty messages in continuous logcat output.
Test: gTest logd_unit_tests --gtest_filter=logd.multiple*
Change-Id: Iad93d3efc6a3938a4b87ccadddbd86626a015d44
If a series of messages arrive from a single source with identical
message content payload, then suppress them and generate a chatty
report. The checking is done on a per log id basis.
This alters the assumption that chatty messages are always at the
oldest entries, they now show up in the middle too. To address this
change in behavior we print the first line, a chatty reference
which internally takes little space, then the last line in the series.
This does not conserve processing time in logd, and certainly has no
impact on the long path of formatting and submitting log messages from
from the source, but it may contribute to memory space and signal to
noise savings under heavy spammy loads.
Test: gTest liblog-unit-tests, logd-unit-tests & logcat-unit-tests
Bug: 33535908
Change-Id: I3160c36d4f4e2f8216f528605a1b3993173f4dec
Move existing fast<str*cmp> templates for general use, pulled from
the implementation used in logd that dealt with speed through cache
locality and subroutine call mitigation. Rename to fastcmp.
Test: logd-benchmarks and based on manual profiling from the past
Bug: 31456426
Change-Id: Ic62f4a437fc3e06ffdeaae73a6f34e197957a6b0
getTag() becomes invalid when entry is dropped because mMsg
disappears to save space; but the per-tag spam filter depends on it
still being valid. Conserve space in LogBufferElement by optimizing
the size of the fields, then add a new mTag field that is set in the
object constructor. Add an isBinary() method.
SideEffects: save 12 bytes/log message overhead on 64-bit.
Test: define DEBUG_CHECK_FOR_STALE_ENTRIES and look for stale entries
Bug: 32247044
Change-Id: Iaa5f416718a92c9e0e6ffd56bd5260d8b908d5c0
Point to log/log.h where necessary, define LOG_TAG where necessary.
Accept that private/android_logger.h is suitable replacement for
log/logger.h and android/log.h.
Correct liblog/README
Effectively a cleanup and controlled select revert of
'system/core: drop or replace log/logger.h' and
'system/core: Replace log/log.h with android/log.h'.
Test: compile
Bug: 30465923
Change-Id: Ic2ad157bad6f5efe2c6af293a73bb753300b17a2
Allows us to mitigate the impact of MAP_PRIVATE and copy on write by
calling android_lookupEventTag_len instead of android_lookupEventTag,
and delaying the copy on write impact to the later. We return a
string length in a supplied location along with the string pointer
with android_lookupEventTag_len(const EventTagMap* map, size_t* len,
int tag). The string is not guaranteed to be nul terminated. Since
android_lookupEventTag() called even once can cause the memory
impact, we will mark it as deprecated, but we currently have no
timeframe for removal since this is a very old interface.
Add an API for __android_log_is_loggable_len() that accepts the non
null terminated content and fixup callers that would gain because the
length is known prior to the call either in the compiler or at
runtime. Tackle transition to android_lookupEventTag_len() and
fixup callers.
On any application that performs logging (eg: com.android.phone)
/proc/<pid>/smaps before:
xxxxxxxxxx-xxxxxxxxxx rw-p 00000000 fd:00 463 /system/etc/event-log-tags
Size: 20 kB
Rss: 20 kB
Pss: 1 kB
Shared_Clean: 0 kB
Shared_Dirty: 20 kB
Private_Clean: 0 kB
Private_Dirty: 0 kB
Referenced: 0 kB
Anonymous: 20 kB
AnonHugePages: 0 kB
Swap: 0 kB
SwapPss: 0 kB
KernelPageSize: 4 kB
MMUPageSize: 4 kB
Locked: 0 kB
VmFlags: rd wr mr mw me ac
/proc/<pid>/smaps after:
xxxxxxxxxx-xxxxxxxxxx rw-p 00000000 fd:00 1773 /system/etc/event-log-tags
Size: 20 kB
Rss: 20 kB
Pss: 1 kB
Shared_Clean: 20 kB (was 0kB)
Shared_Dirty: 0 kB (was 20kB)
Private_Clean: 0 kB
Private_Dirty: 0 kB
Referenced: 20 kB (was 0kB)
Anonymous: 0 kB (was 20kB)
AnonHugePages: 0 kB
Swap: 0 kB
SwapPss: 0 kB
KernelPageSize: 4 kB
MMUPageSize: 4 kB
Locked: 0 kB
VmFlags: rd wr mr mw me ac
Added liblog-unit-tests --gtest_filter=liblog.event_log_tags to
check for Shared_Clean: to not be 0 and Anonymous: to be 0 for
all processes referencing event-log-tags. Which can include multiple
references to /system/etc/event-log-tags and future possible refs to
/data/misc/logd/event-log-tags and /dev/event-log-tags. We want
failure messages to help point to errant code using the deprecated
interface.
This change saves 1/4MB of memory or more on a typical system.
Test: gTest liblog-unit-tests
Bug: 31456426
Change-Id: I9e08e44d9092bd96fe704b5709242e7195281d33
LogReader.h needs to be individually importable.
Fix a few others, drop includes of local includes, let them be
included in source instead and allow headers to be included
alphabetically. Was not a complete audit since goal was to
separate LogReader.h out from the pack.
Bug: 27242723
Change-Id: Ic7759ef90995e5bd285810706af33550c73cf5b5
Adds the uid field to outgoing content for readlog applications.
AID_LOG, AID_ROOT and AID_SYSTEM gain access to the information.
Bug: 25996918
Change-Id: I0254303c19d174cbf5e722c38844be5c54410c85
Although ever present, an increased regression introduced with
commit b6bee33182 (liblog: logd:
support logd.timestamp = monotonic).
A signal handler can interrupt in locked context, if log is written
in the signal handler, we are in deadlock. Block signals while we
are locked. Separate out timestamp lock from is loggable lock to
reduce contention situations. Provide a best-guess response if
lock would fail in timestamp path.
Bug: 25563384
Change-Id: I6dccd6b99ebace1c473c03a785a35c63ed5c6a8a
Simplify table generation by placing the line and header formatting
into each type's (UID, PID, TID) object. Switch to const return
values for the ownership passing functions (*ToName() functions
and methods). Use longer variable names to reduce confusion.
Switch from LINES To NUM for pruned column as that more accurately
reflects what is dropped since one entry can contain several lines.
Bug: 22855208
Change-Id: Ib110dce98a68cf5f844eb30f8a192a1f691eeba2
Discovered that we had a few libc hotspots. Adjust code to generally
reduce or nullify the number of calls to malloc, free, strlen,
strcmp, strncmp, memcmp & strncasecmp. Total gain looks to be about
3% of logd's processing time. malloc still contributes to 3%, but all
others are now total 0.5%.
Bug: 23685592
Change-Id: Ife721121667969260cdb8b055524ae90f5911278
Quick low-risk to resolve possible hash table corruption.
Resolved an unlikely path memory leak.
ToDo: replace lock with nested lock so no lock
helpers are required.
Bug: 22068332
Change-Id: I303ab06608502c7d61d42f111a9c43366f184d0c
- regression in log_strtok_r (part deux) In commit
'logd: fix kernel logline stutter'
2c3b300fd8 we introduced log_strtok_r.
as a replacement for strtok_r that dealt with a problem with
some kernel log messages. Fix is to refine definition of
is_timestamp to not match on patterns like [0], requiring
a single period. Another fix is to refine definition of
is_prio to properly escape non-digit content.
- Missing content because SYSLOG_ACTION_SIZE_BUFFER with added logging
is too short for full read of SYSLOG_ACTION_READ_ALL dropping
initial content. Add a margin for additional 1024 bytes.
- Absolute _first_ log entry has sequence number of 1, which is
specifically dropped, start sequence count at 1 rather than 0.
- Remove trailing space for efficiency.
- If tag exists but no content, trick into kernel logging.
Bug: 21851884
Change-Id: I0867a555a3bca09bbf18d18e75e41dffffe57a23
- Default level when not specified is ANDROID_LOG_VERBOSE
which is inert.
Bug: 20416721
Bug: 19544788
Bug: 17760225
Change-Id: Icc098e53dc47ceaaeb24ec42eb6f61d6430ec2f6
- Cleanup resulting from experience and feedback
- When filtering inside logd, drop any leading expire messages, they
are cluttering up leading edge of tombstones (which filter by pid)
- Increase and introduce EXPIRE_RATELIMIT from 1 to 10 seconds
- Increase EXPIRE_THRESHOLD from 4 to 10 count
- Improve the expire messages from:
logd : uid=1000(system) too chatty comm=com.google.android.phone,
expire 2800 lines
change tag to be more descriptive, and reduce accusatory tone to:
chatty : uid=1000(system) com.google.android.phone expire 2800
lines
- if the UID name forms a prefix for comm name, then drop UID name
Change-Id: Ied7cc04c0ab3ae02167649a0b97378e44ef7b588
- Optional class of statistics for events Tags
- export tagToName from LogBuffer (located in main.cp to address
https://android-review.googlesource.com/#/c/110204)
- Can not handle dropped because getTag() can not work,
will need to be fixed if we start filtering.
Bug: 19608965
Change-Id: I7b90607ca588bf37bab71f19b1570a290e772776
- Report applications UID, TID/PID by name.
- change wording to have an accurate connotation
- drop privilege check since filtered upstream
Bug: 19608965
Bug: 20334069
Bug: 20370119
Change-Id: I2b1c26580b4c2de293874214ff5ae745546f3cca
- internal dropped entries are associated by prune by worst UID
and are applied by UID and by PID
- track dropped entries by rewriting them in place
- merge similar dropped entries together for same UID(implied),
PID and TID so that blame can more clearly be placed
- allow aging of dropped entries by the general backgound pruning
- report individual dropped entries formatted to reader
- add statistics to track dropped entries by UID, the combination
of statistics and dropped logging can track over-the-top Chattiest
clients.
Bug: 19608965
Change-Id: Ibc68480df0c69c55703270cd70c6b26aea165853
- switch to simpler and faster internal sequence number, drops
a syscall overhead on 32-bit platforms.
- add ability to break-out of flushTo loop with filter return -1
allowing in reduction in reader overhead.
Change-Id: Ic5cb2b9afa4d9470153971fc9197b07279e2b79d
Tom Cherry