Skip to content

[#1355]Fix: reject non-Metric types when deserializing algorithm metrics - #1369

Open
NopeCharon wants to merge 12 commits into
Angel-ML:branch-3.4.0from
NopeCharon:fix#1355
Open

NopeCharon wants to merge 12 commits into
Angel-ML:branch-3.4.0from
NopeCharon:fix#1355

Conversation

@NopeCharon

Copy link
Copy Markdown

Pre-submission checklist:

Please check each of these after submitting your pull request:

  • Do you submit an issue before submitting your pull request, each pull request is supposed to connect to an issue. If your do not submit an issue, please submit one at first.
  • Do you connect you pull request to the issue. The title of pull request should start with [Issue ID]
  • Do you assign a code reviewer? If your don't know how to select code reviewers, please refer CODEOWNERS
  • If you pull request is big, it would be hard to merge by the community at once, please contact the TSC member in advance.

What this PR does

Fixes the deserialization part of #1355.

MasterService.setAlgoMetrics hands the bytes from the RPC request to KryoUtils.deserializeAlgoMetric, which called readClassAndObject and cast the result to Metric only afterwards. Two things followed from that:

  • any class named in the payload was instantiated and its fields populated before the cast failed;
  • a forged primitive array header (for example long[] with a length near Integer.MAX_VALUE) made the Master allocate the whole array before reading any element, and the OutOfMemoryError took down the Master.

deserializeAlgoMetric now reads the class header on its own, checks that it is a Metric subclass, and only then reads the object with readObject. Anything else is rejected with a KryoException before an object is created. The check is by type, not a fixed list of LossMetric/ErrorMetric/ObjMetric, so user-defined Metric subclasses keep working.

Because the class and the object are now read in two calls, auto reset is turned off on the thread-local KryoReflectionFactorySupport used here and the graph state is reset in a finally block once the metric has been read. Without this, Kryo would clear its class name table right after readClass, and a metric whose fields refer back to its own class would fail with Unable to find class.

The wire format is unchanged, so the serializing side (MasterClient) is untouched.

Not covered here: the RPC endpoint itself is still unauthenticated. The issue also suggests adding authentication, which is a much larger design change and is better handled separately.

Tests

New KryoUtilsTest (JUnit 4, same as the other core tests):

  • round trip of LossMetric, ErrorMetric and ObjMetric
  • round trip of a null metric
  • a custom metric with a field of its own class, read twice to show the graph state is reset between calls
  • a non-Metric object is rejected with KryoException
  • a forged long[] header with a huge length is rejected with KryoException

How I ran them: I compiled KryoUtils and the test with JDK 8 against the pinned kryo-shaded 4.0.0 and kryo-serializers 0.42 jars, using the released angel-ps-core 3.2.0 jar for the Metric classes. I couldn't run the full Maven build because protoc 2.5.0 wasn't available on my build machine.

  • With this change: all 5 tests pass.
  • With the original deserializeAlgoMetric: the two rejection tests fail with the behaviour described in the issue (ClassCastException for the non-metric object, OutOfMemoryError: Requested array size exceeds VM limit for the forged array).
  • With the type check but without the reset handling: the self-referencing metric test fails.

NopeCharon and others added 11 commits October 24, 2026 18:15
…thm metrics

MasterService.setAlgoMetrics passes the bytes received over RPC to
KryoUtils.deserializeAlgoMetric, which called readClassAndObject and only
cast the result to Metric afterwards. Any class named in the payload was
therefore instantiated and populated first, and a forged primitive array
header with a huge length made the Master allocate the whole array before
reading a single element, crashing it with an OutOfMemoryError.

Read the class header on its own and check that it is a Metric subclass
before reading the object, throwing a KryoException otherwise. Custom
Metric subclasses keep working because the check is by type rather than
a fixed list of the bundled metrics.

Since the class and the object are now read in two calls, auto reset is
disabled on the thread-local Kryo used for deserialization and the graph
state is reset once the whole metric has been read. Otherwise class name
references inside the metric (for example a field of the metric's own
class) could not be resolved.

Add KryoUtilsTest covering round trips of the bundled metrics, a null
metric, a metric that references its own class, and rejection of a
non-metric object and of a forged long[] header.

Signed-off-by: NopeCharon <115794254+NopeCharon@users.noreply.github.com>
…dd standalone PS deployment and stability fixes

Overview
- Move the SONA graph algorithm layer from spark-on-angel-graph to
  spark-on-angel-mllib (package com.tencent.angel.graph.* ->
  com.tencent.angel.spark.ml.graph.algo.*); spark-on-angel-mllib now depends
  on spark-on-angel-graph instead of the reverse.
- Verified end to end on Hadoop 2.10.2 + Spark 3.3.4 (yarn-cluster) with 45
  graph / embedding / ANN examples.

Features
- Graph algorithms: SCC, BatchedCC, CompressCC, CutCC, LPAX, LouvainCPM /
  Density / Repair / Opt, BGE, LINE v2/v3, MetaPath2Vec V1/V2, DocVec,
  EmbeddingPooling, ClusterRank, KCoreDC, KCoreCompress, PPR, WHIndex,
  Intimacy, click-rate features (OneHop, OneHopCom, OneHopWeighted,
  TwoHopCom, LBSBased), edge features, LinkPrediction, PostIndex,
  KCliqueV2, directed triangle / LCC, SSSP / MSSP, IVFHnsw, KMeans-IVFHnsw.
- ML modules: time series (ARIMA, GARCH, Holt-Winters, ...), KD-Tree,
  IsolationForest / LOF / KNN, Apriori / FPGrowth / PrefixSpan, feature
  preprocessing, word segmentation, PMML export, ALS.
- PS: STANDALONE / EMBEDDED deployment modes and attach mode with launch
  scripts under angel-ps/bin; metrics framework; FlatBuffers serialization;
  new graph storage, sampling and label/timestamp PSFs.
- Transport: heartbeats, ordered retry queue, request deadline, bounded
  thread pools, idle connection reclamation, PSF timeout
  (-Dangel.psf.timeout.ms).
- Examples: 70 cluster and 74 local examples, ClusterExampleUtils, MixedLR /
  NFM / PNN json configs, and test data under data/.
- Add RELEASE_NOTES_3.4.0.md and RELEASE_NOTES_3.4.0_CN.md.

Bug fixes
- YarnContainerAllocator: custom ResourceRequest comparator to avoid NPE on
  Hadoop 2.9+/2.10.
- Transport: fix resource leak on fatal/abandoned requests, response vs
  timeout race, no-op Thread.stop(), dispatcher dying on one exception,
  throttled recovery events, ChannelManager2 checker never started, port
  bind races.
- Storage: fix deepClone returning empty maps, dataSize NPE under concurrent
  rehash, silently loading truncated snapshots.
- Graph: wait on psfUpdate futures (LinkPrediction, SLPA, COPRA, KCoreDC,
  LINE, LINE v3); use RangePartitioner for OneHopWeighted and ClusterRank
  matrices; fix empty LINE v3 text output and OneHopWeighted
  ClassCastException.
- Deploy: replace netty-all-4.1.42 / stream-2.7.0 in spark-on-angel-env.sh
  with jars shipped in the release; pin jackson 2.13.4 for Spark 3.3;
  LINEExample only sets master when mode is given.
…n 2 Spark tasks and PS partitions

Verified on YARN cluster mode (2 PS, partitionNum=4, psPartitionNum=4): all 46
algorithms succeed, every non-trivial Spark stage runs > 2 tasks and every PS
matrix has > 2 partitions.

PS partitioning
- RangePartitioner: clamp zero block row/col size to 1 to avoid an infinite loop
  (master hang and RPC timeout) when partitions outnumber rows/cols
- ModelContextUtils / EmbeddingPoolingModel: use ceil division with a minimum of 1
  for block size
- SubSampling/LINEExample: honor psPartitionNum for the freq matrix

Spark parallelism
- Pass minPartitions to textFile/GraphIO.loadString when reading input
  (ClusterExampleUtils, BGE, LINE, LINEV2, LINEV3, Word2vec, DocVec,
  EmbeddingPooling, EGESRemapping, NeighborTabJoinSideInfoTab, ann DataUtils)
- NEModel: sliced saving uses max(executors, spark.default.parallelism) partitions
- RePartition: write partition bounds with partitionNum tasks instead of 1

Correctness with more partitions
- FunctionalEdgeAggregate: compute neighbor degrees globally and join them to
  edges; looking them up within one partition caused ArrayIndexOutOfBounds (-1)
- NeighborTablePartition.testPS: check min(num, size) nodes instead of asserting
  that every partition has at least num nodes
- ANNPSModel / IVFHnswPSModel: skip PS calls for empty query partitions, which
  produced PSF requests with no splits
Verified in Spark local mode: 88 of 89 examples under examples/local succeed
(AngelClientTest needs an external PS address and is out of scope). The YARN
cluster run (46 algorithms, partitionNum=4, psPartitionNum=4) still passes
with these changes.

Library fixes
- DataParser: extend Serializable so its case-class subclasses can be
  deserialized by Spark ("no valid constructor" in OnlineExample)
- OnlineLearner: skip empty micro-batches and empty partitions instead of
  issuing PS pulls with null column indices
- HyperANFPSModel: skip PSF calls for empty node arrays/updates, which
  produced PSF requests with zero splits (HyperDecimal)
- MultiAttrHIndexPSModel.initAttributes: use fold(0L) so an empty RDD returns
  0 instead of throwing "empty collection"

Example fixes
- GBDTExample (local): set storageLevel, which was left null
- SSHopLayoutLocalExample / WHIndexLocalExample: default
  spark.executor.extraJavaOptions to "" when it is not set
- BiGraphAggre / MultiAttrAggregate / MultiAttrHIndex / TopCandidate local
  examples: use the range PS router, since node attribute PSFs
  (GetNodeAttrs) only support range partitioning
…loyment modes

Replace legacy yarn-cluster masters with yarn across SONA examples and align mode documentation.

Update SONA-example to use explicit cluster deploy mode and executor JVM options.

Document local, YARN, STANDALONE, EMBEDDED and Attach deployment in Chinese and English, including dependency conflicts, readiness and lifecycle constraints.
… tests

Implement matrix stream routing, enforce range partitioning for CSR neighbors, and support CSR snapshot save/load with checkpoint recovery regressions.

Propagate local master startup failures, bound startup waits, and clean up failed local clusters.

Exclude conflicting Hadoop 3 client dependencies in local-test and run ScalaTest in a dedicated JVM.

Make test fixtures platform-independent, handle optional LINE embedding paths safely, and restore Word2Vec initialization and loss verification.

Validation: all 14 modules build successfully; PS core 421 passed with 1 SSH-dependent test skipped, graph regressions 3 passed, SONA core 25 passed, and SONA mllib 18 passed.
…gorithms

Reconnect Kubernetes submission, shared Master lifecycle, PS/Worker attempt scheduling, retries and resource cleanup while preserving other deployment modes.

Package Kubernetes image scripts, normalize API URL ports, avoid Hadoop 2/3 classpath conflicts, and isolate native Fabric8 dependencies from Spark using angel.kubernetes.client.classpath.

Resolve late Worker task configuration, pull prediction parameters for initial and tail batches, and propagate GraphRunner prediction failures.

Add Kubernetes factory, pod specification and submission bridge regressions; publish the classified Angel parameter handbook and reference with deployment guidance and validation boundaries.

Validation: all 14 modules build; 16 targeted unit tests pass. Single-node Kubernetes 1.31 E2E covers matrix save/load and PS replacement, native LR train/save/load/predict, and PageRank on Spark 3.3.4 driver/executor pods via attach and direct PS launch. Algorithm outputs checked against independent references; multi-node and all-algorithm coverage remain unvalidated.
…trips

Align Json4s with Spark 3.3 and initialize optional PMML converters only when export is requested; retain explicit PMML export compatibility limits.

Preserve unknown-dimension sparse storage, save columns using actual row storage offsets for hash/range routing, and prevent the model saver dispatcher from busy-spinning.

Parse PSF responses once, complete low-level futures correctly, and retain bounded CSR payload slices with correct pooled-buffer cursor and ownership handling to unblock LDA.

Handle empty neighbor requests, remove time-series UDF closure capture, correct output schemas and graph example context/partition handling, and explicitly stop JSON/offline runner resources.

Parameterize basic PS examples and implement real logistic gradients with LIBSVM index bounds, valid Adam update steps and numerical assertions; add focused regression tests.

Validation: 14-module build and final example rebuild pass; core Java 421 passed/1 skipped, core Scala 14 passed, SONA core 25 passed, SONA mllib 20 passed, graph 3 passed. Original failing Local/YARN cases and KMeans/LDA workflows pass. Independent checks verify model prediction row counts, nonempty mutual-friend recall and Basic LR logloss reduction from 0.693147 to 0.562404. Corrupted old KMeans checkpoints require regeneration; SSH/external PS prerequisites remain unchanged.
…ass loading

Validate and normalize client supplied model paths at the Master save/load RPC boundary: reject empty paths, parent traversal and unsupported file authorities, and optionally confine paths to configured roots.

Prevent model saves from recursively deleting arbitrary existing directories: the final commit only replaces genuine Angel model directories (or an explicitly enabled overwrite), writes a job ownership marker, and verifies it before loading. Legacy model directories without a marker remain readable with a warning.

Restrict dynamic class loading: RPC protocol names must resolve to VersionedProtocol interfaces and model format classes to concrete MatrixFormat implementations inside the Angel packages, with configuration based allow lists for extensions.

Bind each PS attempt to its first registered location so heartbeat re-registration stays idempotent but conflicting addresses are rejected; add a state-machine guard and an optional caller source address check.

Pin Jackson core/databind/annotations to the Spark 3.3 runtime versions to remove the test classpath conflict introduced by Json4s 3.7.0-M11, and add ModelSecurityTest covering traversal, safe overwrite, ownership markers, class allow lists and registration conflicts.

Validation: angel-ps-core compiles and jars; 12 new security tests pass; core Java 386 passed/9 skipped with 0 failures and 0 errors, core Scala 14 passed; Local Spark ML train+predict and JsonRunner train/save, predict/load and re-save all pass with the ownership marker written; YARN JsonRunner train succeeds; git diff --check is clean. Master RPC still has no cryptographic authentication; strict allowed roots, source address checking and full credential support remain deployment/protocol hardening work.
Drop 23 .fbs schema files from angel-ps/core/src/main/fbs that are not
part of the current build or runtime path:

- 15 optimizer-specific schemas (Adadelta, Adagrad, Adam, FTRL, Lion,
  Momentum, Nadam, RMSprop, SGD, etc.)
- shared gradient, training, quantization, hyper-parameter, init, common
  and frequency-stat schemas that were either referenced only from the
  removed set or had no references at all

Keep weight_message.fbs and stats_param.fbs. They are the source of the six
checked-in FlatBuffers generated classes under
com.tencent.angel.flatbuffers.dollar.tensor, and WeightMessage is used by
AdsEmbeddingOutputFormat for embedding model output. stats_param.fbs is
still included by weight_message.fbs and backs TaskStats.

Validation:
- no Maven plugin or script runs flatc over src/main/fbs; angel-ps/core/pom.xml
  only declares the flatbuffers-java 23.5.26 runtime for the checked-in stubs
- git grep finds no .fbs references outside src/main/fbs and no Java/Scala
  references to the removed message types
- all six generated Java stubs remain; the remaining include graph is
  weight_message.fbs -> stats_param.fbs with no dangling includes
- git status shows exactly 23 deletions and no other tracked changes

.fbs files are not compiled by Maven, so no code-generation or test rerun
was required for this cleanup. Untracked validation artifacts were excluded
from the commit.
Signed-off-by: NopeCharon <115794254+NopeCharon@users.noreply.github.com>
Signed-off-by: NopeCharon <115794254+NopeCharon@users.noreply.github.com>

# Conflicts:
#	angel-ps/core/src/main/java/com/tencent/angel/utils/KryoUtils.java
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants