Repository navigation
[#1355]Fix: reject non-Metric types when deserializing algorithm metrics - #1369
Open
NopeCharon wants to merge 12 commits into
Open
NopeCharon wants to merge 12 commits into
NopeCharon wants to merge 12 commits into
Conversation
…thm metrics MasterService.setAlgoMetrics passes the bytes received over RPC to KryoUtils.deserializeAlgoMetric, which called readClassAndObject and only cast the result to Metric afterwards. Any class named in the payload was therefore instantiated and populated first, and a forged primitive array header with a huge length made the Master allocate the whole array before reading a single element, crashing it with an OutOfMemoryError. Read the class header on its own and check that it is a Metric subclass before reading the object, throwing a KryoException otherwise. Custom Metric subclasses keep working because the check is by type rather than a fixed list of the bundled metrics. Since the class and the object are now read in two calls, auto reset is disabled on the thread-local Kryo used for deserialization and the graph state is reset once the whole metric has been read. Otherwise class name references inside the metric (for example a field of the metric's own class) could not be resolved. Add KryoUtilsTest covering round trips of the bundled metrics, a null metric, a metric that references its own class, and rejection of a non-metric object and of a forged long[] header. Signed-off-by: NopeCharon <115794254+NopeCharon@users.noreply.github.com>
…dd standalone PS deployment and stability fixes Overview - Move the SONA graph algorithm layer from spark-on-angel-graph to spark-on-angel-mllib (package com.tencent.angel.graph.* -> com.tencent.angel.spark.ml.graph.algo.*); spark-on-angel-mllib now depends on spark-on-angel-graph instead of the reverse. - Verified end to end on Hadoop 2.10.2 + Spark 3.3.4 (yarn-cluster) with 45 graph / embedding / ANN examples. Features - Graph algorithms: SCC, BatchedCC, CompressCC, CutCC, LPAX, LouvainCPM / Density / Repair / Opt, BGE, LINE v2/v3, MetaPath2Vec V1/V2, DocVec, EmbeddingPooling, ClusterRank, KCoreDC, KCoreCompress, PPR, WHIndex, Intimacy, click-rate features (OneHop, OneHopCom, OneHopWeighted, TwoHopCom, LBSBased), edge features, LinkPrediction, PostIndex, KCliqueV2, directed triangle / LCC, SSSP / MSSP, IVFHnsw, KMeans-IVFHnsw. - ML modules: time series (ARIMA, GARCH, Holt-Winters, ...), KD-Tree, IsolationForest / LOF / KNN, Apriori / FPGrowth / PrefixSpan, feature preprocessing, word segmentation, PMML export, ALS. - PS: STANDALONE / EMBEDDED deployment modes and attach mode with launch scripts under angel-ps/bin; metrics framework; FlatBuffers serialization; new graph storage, sampling and label/timestamp PSFs. - Transport: heartbeats, ordered retry queue, request deadline, bounded thread pools, idle connection reclamation, PSF timeout (-Dangel.psf.timeout.ms). - Examples: 70 cluster and 74 local examples, ClusterExampleUtils, MixedLR / NFM / PNN json configs, and test data under data/. - Add RELEASE_NOTES_3.4.0.md and RELEASE_NOTES_3.4.0_CN.md. Bug fixes - YarnContainerAllocator: custom ResourceRequest comparator to avoid NPE on Hadoop 2.9+/2.10. - Transport: fix resource leak on fatal/abandoned requests, response vs timeout race, no-op Thread.stop(), dispatcher dying on one exception, throttled recovery events, ChannelManager2 checker never started, port bind races. - Storage: fix deepClone returning empty maps, dataSize NPE under concurrent rehash, silently loading truncated snapshots. - Graph: wait on psfUpdate futures (LinkPrediction, SLPA, COPRA, KCoreDC, LINE, LINE v3); use RangePartitioner for OneHopWeighted and ClusterRank matrices; fix empty LINE v3 text output and OneHopWeighted ClassCastException. - Deploy: replace netty-all-4.1.42 / stream-2.7.0 in spark-on-angel-env.sh with jars shipped in the release; pin jackson 2.13.4 for Spark 3.3; LINEExample only sets master when mode is given.
…n 2 Spark tasks and PS partitions Verified on YARN cluster mode (2 PS, partitionNum=4, psPartitionNum=4): all 46 algorithms succeed, every non-trivial Spark stage runs > 2 tasks and every PS matrix has > 2 partitions. PS partitioning - RangePartitioner: clamp zero block row/col size to 1 to avoid an infinite loop (master hang and RPC timeout) when partitions outnumber rows/cols - ModelContextUtils / EmbeddingPoolingModel: use ceil division with a minimum of 1 for block size - SubSampling/LINEExample: honor psPartitionNum for the freq matrix Spark parallelism - Pass minPartitions to textFile/GraphIO.loadString when reading input (ClusterExampleUtils, BGE, LINE, LINEV2, LINEV3, Word2vec, DocVec, EmbeddingPooling, EGESRemapping, NeighborTabJoinSideInfoTab, ann DataUtils) - NEModel: sliced saving uses max(executors, spark.default.parallelism) partitions - RePartition: write partition bounds with partitionNum tasks instead of 1 Correctness with more partitions - FunctionalEdgeAggregate: compute neighbor degrees globally and join them to edges; looking them up within one partition caused ArrayIndexOutOfBounds (-1) - NeighborTablePartition.testPS: check min(num, size) nodes instead of asserting that every partition has at least num nodes - ANNPSModel / IVFHnswPSModel: skip PS calls for empty query partitions, which produced PSF requests with no splits
Verified in Spark local mode: 88 of 89 examples under examples/local succeed
(AngelClientTest needs an external PS address and is out of scope). The YARN
cluster run (46 algorithms, partitionNum=4, psPartitionNum=4) still passes
with these changes.
Library fixes
- DataParser: extend Serializable so its case-class subclasses can be
deserialized by Spark ("no valid constructor" in OnlineExample)
- OnlineLearner: skip empty micro-batches and empty partitions instead of
issuing PS pulls with null column indices
- HyperANFPSModel: skip PSF calls for empty node arrays/updates, which
produced PSF requests with zero splits (HyperDecimal)
- MultiAttrHIndexPSModel.initAttributes: use fold(0L) so an empty RDD returns
0 instead of throwing "empty collection"
Example fixes
- GBDTExample (local): set storageLevel, which was left null
- SSHopLayoutLocalExample / WHIndexLocalExample: default
spark.executor.extraJavaOptions to "" when it is not set
- BiGraphAggre / MultiAttrAggregate / MultiAttrHIndex / TopCandidate local
examples: use the range PS router, since node attribute PSFs
(GetNodeAttrs) only support range partitioning
…loyment modes Replace legacy yarn-cluster masters with yarn across SONA examples and align mode documentation. Update SONA-example to use explicit cluster deploy mode and executor JVM options. Document local, YARN, STANDALONE, EMBEDDED and Attach deployment in Chinese and English, including dependency conflicts, readiness and lifecycle constraints.
… tests Implement matrix stream routing, enforce range partitioning for CSR neighbors, and support CSR snapshot save/load with checkpoint recovery regressions. Propagate local master startup failures, bound startup waits, and clean up failed local clusters. Exclude conflicting Hadoop 3 client dependencies in local-test and run ScalaTest in a dedicated JVM. Make test fixtures platform-independent, handle optional LINE embedding paths safely, and restore Word2Vec initialization and loss verification. Validation: all 14 modules build successfully; PS core 421 passed with 1 SSH-dependent test skipped, graph regressions 3 passed, SONA core 25 passed, and SONA mllib 18 passed.
…gorithms Reconnect Kubernetes submission, shared Master lifecycle, PS/Worker attempt scheduling, retries and resource cleanup while preserving other deployment modes. Package Kubernetes image scripts, normalize API URL ports, avoid Hadoop 2/3 classpath conflicts, and isolate native Fabric8 dependencies from Spark using angel.kubernetes.client.classpath. Resolve late Worker task configuration, pull prediction parameters for initial and tail batches, and propagate GraphRunner prediction failures. Add Kubernetes factory, pod specification and submission bridge regressions; publish the classified Angel parameter handbook and reference with deployment guidance and validation boundaries. Validation: all 14 modules build; 16 targeted unit tests pass. Single-node Kubernetes 1.31 E2E covers matrix save/load and PS replacement, native LR train/save/load/predict, and PageRank on Spark 3.3.4 driver/executor pods via attach and direct PS launch. Algorithm outputs checked against independent references; multi-node and all-algorithm coverage remain unvalidated.
…trips Align Json4s with Spark 3.3 and initialize optional PMML converters only when export is requested; retain explicit PMML export compatibility limits. Preserve unknown-dimension sparse storage, save columns using actual row storage offsets for hash/range routing, and prevent the model saver dispatcher from busy-spinning. Parse PSF responses once, complete low-level futures correctly, and retain bounded CSR payload slices with correct pooled-buffer cursor and ownership handling to unblock LDA. Handle empty neighbor requests, remove time-series UDF closure capture, correct output schemas and graph example context/partition handling, and explicitly stop JSON/offline runner resources. Parameterize basic PS examples and implement real logistic gradients with LIBSVM index bounds, valid Adam update steps and numerical assertions; add focused regression tests. Validation: 14-module build and final example rebuild pass; core Java 421 passed/1 skipped, core Scala 14 passed, SONA core 25 passed, SONA mllib 20 passed, graph 3 passed. Original failing Local/YARN cases and KMeans/LDA workflows pass. Independent checks verify model prediction row counts, nonempty mutual-friend recall and Basic LR logloss reduction from 0.693147 to 0.562404. Corrupted old KMeans checkpoints require regeneration; SSH/external PS prerequisites remain unchanged.
…ass loading Validate and normalize client supplied model paths at the Master save/load RPC boundary: reject empty paths, parent traversal and unsupported file authorities, and optionally confine paths to configured roots. Prevent model saves from recursively deleting arbitrary existing directories: the final commit only replaces genuine Angel model directories (or an explicitly enabled overwrite), writes a job ownership marker, and verifies it before loading. Legacy model directories without a marker remain readable with a warning. Restrict dynamic class loading: RPC protocol names must resolve to VersionedProtocol interfaces and model format classes to concrete MatrixFormat implementations inside the Angel packages, with configuration based allow lists for extensions. Bind each PS attempt to its first registered location so heartbeat re-registration stays idempotent but conflicting addresses are rejected; add a state-machine guard and an optional caller source address check. Pin Jackson core/databind/annotations to the Spark 3.3 runtime versions to remove the test classpath conflict introduced by Json4s 3.7.0-M11, and add ModelSecurityTest covering traversal, safe overwrite, ownership markers, class allow lists and registration conflicts. Validation: angel-ps-core compiles and jars; 12 new security tests pass; core Java 386 passed/9 skipped with 0 failures and 0 errors, core Scala 14 passed; Local Spark ML train+predict and JsonRunner train/save, predict/load and re-save all pass with the ownership marker written; YARN JsonRunner train succeeds; git diff --check is clean. Master RPC still has no cryptographic authentication; strict allowed roots, source address checking and full credential support remain deployment/protocol hardening work.
Drop 23 .fbs schema files from angel-ps/core/src/main/fbs that are not part of the current build or runtime path: - 15 optimizer-specific schemas (Adadelta, Adagrad, Adam, FTRL, Lion, Momentum, Nadam, RMSprop, SGD, etc.) - shared gradient, training, quantization, hyper-parameter, init, common and frequency-stat schemas that were either referenced only from the removed set or had no references at all Keep weight_message.fbs and stats_param.fbs. They are the source of the six checked-in FlatBuffers generated classes under com.tencent.angel.flatbuffers.dollar.tensor, and WeightMessage is used by AdsEmbeddingOutputFormat for embedding model output. stats_param.fbs is still included by weight_message.fbs and backs TaskStats. Validation: - no Maven plugin or script runs flatc over src/main/fbs; angel-ps/core/pom.xml only declares the flatbuffers-java 23.5.26 runtime for the checked-in stubs - git grep finds no .fbs references outside src/main/fbs and no Java/Scala references to the removed message types - all six generated Java stubs remain; the remaining include graph is weight_message.fbs -> stats_param.fbs with no dangling includes - git status shows exactly 23 deletions and no other tracked changes .fbs files are not compiled by Maven, so no code-generation or test rerun was required for this cleanup. Untracked validation artifacts were excluded from the commit.
Signed-off-by: NopeCharon <115794254+NopeCharon@users.noreply.github.com>
Signed-off-by: NopeCharon <115794254+NopeCharon@users.noreply.github.com> # Conflicts: # angel-ps/core/src/main/java/com/tencent/angel/utils/KryoUtils.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pre-submission checklist:
Please check each of these after submitting your pull request:
[Issue ID]What this PR does
Fixes the deserialization part of #1355.
MasterService.setAlgoMetricshands the bytes from the RPC request toKryoUtils.deserializeAlgoMetric, which calledreadClassAndObjectand cast the result toMetriconly afterwards. Two things followed from that:long[]with a length nearInteger.MAX_VALUE) made the Master allocate the whole array before reading any element, and theOutOfMemoryErrortook down the Master.deserializeAlgoMetricnow reads the class header on its own, checks that it is aMetricsubclass, and only then reads the object withreadObject. Anything else is rejected with aKryoExceptionbefore an object is created. The check is by type, not a fixed list ofLossMetric/ErrorMetric/ObjMetric, so user-definedMetricsubclasses keep working.Because the class and the object are now read in two calls, auto reset is turned off on the thread-local
KryoReflectionFactorySupportused here and the graph state is reset in afinallyblock once the metric has been read. Without this, Kryo would clear its class name table right afterreadClass, and a metric whose fields refer back to its own class would fail withUnable to find class.The wire format is unchanged, so the serializing side (
MasterClient) is untouched.Not covered here: the RPC endpoint itself is still unauthenticated. The issue also suggests adding authentication, which is a much larger design change and is better handled separately.
Tests
New
KryoUtilsTest(JUnit 4, same as the other core tests):LossMetric,ErrorMetricandObjMetricnullmetricMetricobject is rejected withKryoExceptionlong[]header with a huge length is rejected withKryoExceptionHow I ran them: I compiled
KryoUtilsand the test with JDK 8 against the pinnedkryo-shaded4.0.0 andkryo-serializers0.42 jars, using the releasedangel-ps-core3.2.0 jar for theMetricclasses. I couldn't run the full Maven build becauseprotoc2.5.0 wasn't available on my build machine.deserializeAlgoMetric: the two rejection tests fail with the behaviour described in the issue (ClassCastExceptionfor the non-metric object,OutOfMemoryError: Requested array size exceeds VM limitfor the forged array).