Skip to content

Commit c6dfdb8

Browse files
Merge pull request #160 from Countly/sbs-health-check
sbs and hc
2 parents 64516be + 117af62 commit c6dfdb8

16 files changed

Lines changed: 1798 additions & 77 deletions

‎CHANGELOG.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,17 @@
1515
- Sends the previous user's open view duration and unsaved custom property changes under the previous ID before switching
1616
- Ignores remote config values that were still downloading for the previous device ID. The "fetch_remote_config" callback then gets an error
1717
- ! Minor breaking change ! In a cluster, a worker now asks the main process once whether it runs the SDK, and the main process answers. These messages have a "cly" property, and your own "message" handlers in the main process and in workers receive them as well
18+
- ! Minor breaking change ! Added SDK behavior settings. The SDK now fetches the settings set in the dashboard's SDK Manager at start, every 4 hours (or at the refresh interval set there) and after a device ID change, and follows them:
19+
- The settings can turn off all tracking, all sending, or sessions, views, custom events, crashes and location one by one, change the queue, batch and length limits, and turn the consent requirement on or off, also one set with "require_consent", as on Android and Web. Without that setting the "require_consent" value applies
20+
- A setting set in SDK Manager takes precedence over the matching init option. A setting removed there goes back to the init option, or else to the default
21+
- The fetched settings are stored and used from the next start on, also before the server answers. The new init option "behavior_settings" provides settings, in the format SDK Manager exports, for the time before the first successful fetch
22+
- While tracking is turned off nothing new is recorded, including consent changes and "request" calls, and data recorded before is still sent. While sending is turned off data is kept and sent once sending is turned on again, "flush" resolves false right away and "fetch_remote_config" calls its callback with an error
23+
- The settings request is sent also when consent is required and not given, and while sending is turned off, so that changes reach the SDK. Set the new init option "disable_sdk_behavior_settings_updates" to not fetch the settings; stored and provided settings still apply
24+
- After a device ID change with merge, the settings are fetched once the merge request is delivered
25+
- ! Minor breaking change ! Added a health check. Once per start, after the settings fetch, the SDK sends the number of errors and warnings it logged (also when logging is off), the status code and the first 1000 characters of the answer of the last request the server refused, and how often sending paused, together with the app version. The numbers are kept until the server confirms them. The health check is sent also when consent is required and not given, and not while sending is turned off. Set the new init option "disable_health_check" to not send it
26+
- ! Minor breaking change ! After a request the server answered slowly (in 10 seconds or more by default), the SDK now pauses sending for 60 seconds when the queue is at most half full and the request was recorded within the last 24 hours, so that a busy server can recover. SDK Manager can change these values, and "flush" does not wait for the pause. Set the new init option "disable_backoff_mechanism" to never pause
27+
- SDK Manager can now set a maximum age for queued requests. Older requests are then dropped instead of sent, and a "flush" call waiting for them resolves false. This is off by default
28+
- A missing storage file, which is normal before the SDK first writes it, is no longer logged as a warning
1829
- The npm package now contains only the SDK code, README, LICENSE and CHANGELOG. Tests, examples and repository configuration files are no longer included
1930
- Added a new method "flush" that sends all recorded data right away and tells when it is delivered, for example before a script ends or in a shutdown handler before "process.exit()"
2031
- Added a new init time flag "standalone_worker" that lets a cluster worker send its own data when the main process does not run the SDK, for example in PM2 cluster mode:
@@ -27,6 +38,7 @@
2738
- Fixed a bug where a request was removed from the stored queue before the server confirmed it, so it was lost if the process ended meanwhile
2839
- Fixed a bug where creating a CountlyBulk instance changed where, or whether, the SDK and other CountlyBulk instances stored their data, and where two CountlyBulk instances could send each other's data under the wrong app
2940
- Fixed a bug where removing consent was never reported to the server
41+
- Fixed a bug where a consent change made less than a second before a device ID change was reported under the new device ID or not at all, and where a consent change made right before "flush" and "process.exit()" was lost. Such a change is now sent right away, under the device ID of the user who made it
3042
- Fixed a bug where consent changes of CountlyBulk users were never sent
3143
- Fixed a bug where a BigInt value or an object that refers to itself in event, view, user property or crash data crashed the app. BigInt values are now sent as text and self references are left out
3244
- Fixed a bug where "change_id" with an empty value stopped all tracking. Such values are now ignored with a warning, and an empty device ID stored by an older version is ignored at start
@@ -35,6 +47,7 @@
3547
- Fixed a bug where "request" cut every value to 256 characters, which broke raw event and user payloads. Raw requests now also count towards the request queue size
3648
- Fixed a bug where requests containing an apostrophe failed the checksum check of Countly 26.01 and newer servers when "salt" is set. The characters ! ' ( ) * are now percent-encoded
3749
- Fixed a bug where a value containing half of an emoji, for example a device ID cut in the middle of an emoji, could never be sent and stopped all further sending, also after a restart. Such half characters are now sent as the replacement character "�"
50+
- Fixed a bug where the key length and segmentation limits also cut Countly's own events, so the server no longer recognized them. For example, with a key length limit of 8 a view was sent as "[CLY]_vi". Views, star ratings, feedback and other internal events now keep their key and segmentation; the segmentation given to "track_view" is still limited
3851
- Fixed a bug where a CountlyBulk user created with "ip_address" threw an error on most calls
3952
- Fixed a bug where giving consent to a CountlyBulk user after a "begin_session" or "report_view" call that waited for consent crashed the app. The waiting call is now sent
4053
- Fixed a bug where the CountlyBulk "start" callback reported an empty queue while data was still waiting for a retry or being sent

‎lib/countly-behavior.js‎

Lines changed: 222 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,222 @@
1+
/**
2+
* SDK behavior settings: checks the settings the server sends (from SDK Manager) or the developer provides,
3+
* and decides the value of each setting the SDK uses.
4+
* @module lib/countly-behavior
5+
*/
6+
7+
/**
8+
* Settings the Node.js SDK follows, with the kind of value each one takes:
9+
* "switch" true or false; "count" a whole number of at least 1; "zeroOrMore" a whole number of at least 0;
10+
* "fraction" a number above 0 and at most 1
11+
*/
12+
var SUPPORTED = {
13+
tracking: "switch",
14+
networking: "switch",
15+
st: "switch",
16+
vt: "switch",
17+
cet: "switch",
18+
crt: "switch",
19+
lt: "switch",
20+
cr: "switch",
21+
bom: "switch",
22+
rqs: "count",
23+
eqs: "count",
24+
sui: "count",
25+
lkl: "count",
26+
lvs: "count",
27+
lsv: "count",
28+
lbc: "count",
29+
ltlpt: "count",
30+
ltl: "count",
31+
scui: "count",
32+
bom_at: "count",
33+
bom_ra: "count",
34+
bom_d: "count",
35+
dort: "zeroOrMore",
36+
bom_rqp: "fraction",
37+
};
38+
39+
/**
40+
* Values used when neither the server nor the developer gives one
41+
*/
42+
var DEFAULTS = {
43+
tracking: true,
44+
networking: true,
45+
st: true,
46+
vt: true,
47+
cet: true,
48+
crt: true,
49+
lt: true,
50+
cr: false,
51+
bom: true,
52+
scui: 4,
53+
dort: 0,
54+
bom_at: 10,
55+
bom_rqp: 0.5,
56+
bom_ra: 24,
57+
bom_d: 60,
58+
};
59+
60+
/**
61+
* Longest settings refresh interval in hours. A Node.js timer cannot wait longer than about 24.8 days, and a
62+
* longer value would make it fire at once, over and over.
63+
*/
64+
var MAX_REFRESH_HOURS = 596;
65+
66+
/**
67+
* Tells whether a value is a plain object (not null, not an array)
68+
* @param {*} value - value to check
69+
* @returns {Boolean} true for a plain object
70+
*/
71+
function isPlainObject(value) {
72+
return Object.prototype.toString.call(value) === "[object Object]";
73+
}
74+
75+
/**
76+
* Tells whether a value fits the kind a setting takes
77+
* @param {String} kind - "switch", "count", "zeroOrMore" or "fraction"
78+
* @param {*} value - value from the server or the developer
79+
* @returns {Boolean} true if the value can be used
80+
*/
81+
function isValidValue(kind, value) {
82+
if (kind === "switch") {
83+
return typeof value === "boolean";
84+
}
85+
if (typeof value !== "number" || !Number.isFinite(value)) {
86+
return false;
87+
}
88+
if (kind === "count") {
89+
return Math.floor(value) === value && value >= 1;
90+
}
91+
if (kind === "zeroOrMore") {
92+
return Math.floor(value) === value && value >= 0;
93+
}
94+
return value > 0 && value <= 1;
95+
}
96+
97+
/**
98+
* Keeps only the supported settings with a usable value. Other keys (for example the dashboard's
99+
* "bom_preset", or settings for other platforms) are left out, and an unusable value is left out with a
100+
* warning, so the developer value or the default stays in use for it.
101+
* @param {Object} config - the "c" object of a settings answer
102+
* @param {Function} warn - called with a message for each left out value
103+
* @returns {Object} the usable settings
104+
*/
105+
function normalize(config, warn) {
106+
var result = {};
107+
Object.keys(config).forEach((key) => {
108+
if (!Object.prototype.hasOwnProperty.call(SUPPORTED, key)) {
109+
return;
110+
}
111+
if (isValidValue(SUPPORTED[key], config[key])) {
112+
result[key] = key === "scui" ? Math.min(config[key], MAX_REFRESH_HOURS) : config[key];
113+
}
114+
else if (typeof warn === "function") {
115+
warn(`The behavior setting [${key}] has an unusable value [${JSON.stringify(config[key])}], it is ignored.`);
116+
}
117+
});
118+
return result;
119+
}
120+
121+
/**
122+
* Reads the server's answer to a settings request. The answer must be a JSON object with a "c" object;
123+
* other top-level keys (v, t, lg, ct and future ones) are ignored, so both the v1 and the v2 format work.
124+
* @param {String} responseText - the answer body
125+
* @param {Function} warn - called with a message for each problem
126+
* @returns {Object|null} the usable settings, or null when the answer cannot be used
127+
*/
128+
function parseAnswer(responseText, warn) {
129+
var answer;
130+
try {
131+
answer = JSON.parse(responseText);
132+
}
133+
catch (ex) {
134+
return null;
135+
}
136+
if (!isPlainObject(answer) || !isPlainObject(answer.c)) {
137+
return null;
138+
}
139+
return normalize(answer.c, warn);
140+
}
141+
142+
/**
143+
* Reads the settings a developer provides with the "behavior_settings" init option, in the same format the
144+
* server sends: an object with a "c" object
145+
* @param {*} provided - the init option value
146+
* @param {Function} warn - called with a message for each problem
147+
* @returns {Object|null} the usable settings, or null when none are provided or the format is wrong
148+
*/
149+
function parseProvided(provided, warn) {
150+
if (typeof provided === "undefined" || provided === null) {
151+
return null;
152+
}
153+
if (!isPlainObject(provided) || !isPlainObject(provided.c)) {
154+
if (typeof warn === "function") {
155+
warn("behavior_settings must be an object with a \"c\" object, as SDK Manager exports it. It is ignored.");
156+
}
157+
return null;
158+
}
159+
return normalize(provided.c, warn);
160+
}
161+
162+
/**
163+
* Reads settings stored by an earlier run
164+
* @param {*} stored - the stored value
165+
* @returns {Object|null} the usable settings, or null when nothing usable is stored
166+
*/
167+
function parseStored(stored) {
168+
if (!isPlainObject(stored)) {
169+
return null;
170+
}
171+
return normalize(stored);
172+
}
173+
174+
/**
175+
* Decides the value of a setting: the server's (or the stored or provided) value when there is one, else
176+
* the developer's value, else the default
177+
* @param {Object} settings - settings in use
178+
* @param {String} key - setting key
179+
* @param {*=} developerValue - value set with an init option, undefined when none
180+
* @returns {*} the value to use
181+
*/
182+
function valueOf(settings, key, developerValue) {
183+
if (settings && Object.prototype.hasOwnProperty.call(settings, key)) {
184+
return settings[key];
185+
}
186+
if (typeof developerValue !== "undefined") {
187+
return developerValue;
188+
}
189+
return DEFAULTS[key];
190+
}
191+
192+
/**
193+
* Decides whether sending must pause after a successful request, following the backoff settings: the
194+
* request took at least bom_at seconds, the queue is at most bom_rqp of its size, and the request was
195+
* recorded at most bom_ra hours ago
196+
* @param {Object} input - measurements and settings
197+
* @param {Number} input.durationSeconds - how long the request took
198+
* @param {Number} input.queueLength - requests left in the queue
199+
* @param {Number} input.queueSize - maximum queue size
200+
* @param {Number} input.ageHours - hours since the request was recorded
201+
* @param {Number} input.bom_at - response time threshold in seconds
202+
* @param {Number} input.bom_rqp - queue fill threshold as a fraction of the queue size
203+
* @param {Number} input.bom_ra - request age threshold in hours
204+
* @returns {Boolean} true when sending must pause
205+
*/
206+
function shouldBackOff(input) {
207+
return input.durationSeconds >= input.bom_at
208+
&& input.queueLength <= input.queueSize * input.bom_rqp
209+
&& input.ageHours <= input.bom_ra;
210+
}
211+
212+
module.exports = {
213+
SUPPORTED,
214+
DEFAULTS,
215+
MAX_REFRESH_HOURS,
216+
normalize,
217+
parseAnswer,
218+
parseProvided,
219+
parseStored,
220+
valueOf,
221+
shouldBackOff,
222+
};

‎lib/countly-common.js‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,8 @@ var cc = {
6262
debugBulk: false,
6363
debugBulkUser: false,
6464
warnedOnce: {},
65+
// called with the level of every error and warning log line, also when logging is off; the SDK counts them for its health check
66+
onErrorOrWarning: null,
6567
/**
6668
* Prints a warning once per process, also when logging is off. Used only for setups in which the SDK
6769
* cannot send or keep data at all, so integrators learn about them without turning on debug logs.
@@ -326,6 +328,9 @@ var cc = {
326328
* @param {string} message - any string message
327329
*/
328330
log: function log(level, message, ...args) {
331+
if (typeof this.onErrorOrWarning === "function" && (level === this.logLevelEnums.ERROR || level === this.logLevelEnums.WARNING)) {
332+
this.onErrorOrWarning(level);
333+
}
329334
if ((this.debug || this.debugBulk || this.debugBulkUser) && typeof console !== "undefined") {
330335
if (args[0] instanceof Error) {
331336
args[0] = args[0].stack || String(args[0]);

‎lib/countly-storage.js‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -440,7 +440,8 @@ function createStorage() {
440440
}
441441

442442
/**
443-
* Read value from file
443+
* Read value from file. A missing file is normal before the first write, so only other read errors are
444+
* logged as warnings (the health check counts warnings).
444445
* @param {String} key - key for file
445446
* @returns {varies} value in file
446447
*/
@@ -451,7 +452,12 @@ function createStorage() {
451452
data = fs.readFileSync(file, 'utf8');
452453
}
453454
catch (ex) {
454-
cc.log(cc.logLevelEnums.WARNING, `readFile, File not found for key: [${key}]. Returning null.`);
455+
if (ex && ex.code === "ENOENT") {
456+
cc.log(cc.logLevelEnums.DEBUG, `readFile, File not found for key: [${key}]. Returning null.`);
457+
}
458+
else {
459+
cc.log(cc.logLevelEnums.WARNING, `readFile, Could not read the file for key: [${key}]. Error: [${ex}]. Returning null.`);
460+
}
455461
return null;
456462
}
457463

@@ -761,6 +767,7 @@ module.exports = {
761767
storeRemove: defaultStorage.storeRemove,
762768
writeFile: defaultStorage.writeFile,
763769
forceStore: defaultStorage.forceStore,
770+
flushSync: defaultStorage.flushSync,
764771
getStoragePath: defaultStorage.getStoragePath,
765772
setStoragePath: defaultStorage.setStoragePath,
766773
resetStorage: defaultStorage.resetStorage,

0 commit comments

Comments
 (0)